Poor data quality – and a lack of ability to integrate data from multiple sources – is hurting organisations at all levels. While an absence of information may be frustrating, providing users with access to information that is simply wrong can have calamitous consequences.

At a recent Information Age debate on data management, the IT director of an industrial components manufacturer recounted the embarrassment and annoyance his company had suffered having billed one customer £500,000 for a piece of equipment that was still sitting in its own warehouse. "It looked to be a good bit of business at the time – until the customer started screaming," he said.

For many organisations, poor data quality results from creating multiple records for a single piece of data – and then doing nothing to rationalise that. To illustrate that, one IT director from the hospitality industry told of his attempts to clean the company's stock data: an audit showed that the organisation seemed to have 100 different baguettes on offer because of the different descriptions users had been allowed to create. In fact, it only sells three different types, he explained.

The extent of the problem is so severe that analyst company Gartner predicts that through 2007, more than half of all data warehouse projects – where data has been collated specifically with the purpose of applying detailed, business-critical analysis to it – will have limited acceptance, or worse, be deemed outright failures, as a direct result of a lack of attention to data quality issues.

Such situations have not been helped by mergers and acquisitions. As companies come together, they produce a patchwork infrastructure, with different units running separate – and potentially incompatible – data repositories and warehouses.

But it is not just those businesses that have had to cope with M&A that find themselves with ‘stove pipes' of data: often departmental independence is sufficient to foster the creation of different databases and the zealous protection of those as divisional assets.

"Poor data quality costs business money whichever way you look at it – failure to retain customers, lost productivity, faulty business decisions and an inability to achieve results from expensive investments in technology and infrastructure," says Jeremy Duggan, northern European sales director at Ascential (now part of IBM's information management unit) which sponsored the lunch.

Banking giant Lloyds TSB put a figure on poor data quality, estimating the problem cost the business some £200 million to resolve.

But whereas some businesses were prepared to live with the pain of having poor quality data, increasingly stringent financial regulations are imposing greater discipline. "Proving compliance can be extremely difficult if the data your decisions rest on is inaccurate," says Duggan. The Sarbanes-Oxley regulations from the US have jolted many business executives out of their apathy.

The assistant vice president of an international financial services group explained it had instigated a company wide review of how data was stored and managed across the globe: "We decided that data had to be of uniform quality across the organisation – that was a big wake-up call for some areas."