In information management circles it has become fashionable to quote Oscar Wilde: “It is a very sad thing that nowadays there is so little useless information.”

The observation is an apt one because it captures the nature of the problem that technology has helped to create: modern business is overrun with the need to manage information that it might once have regarded as practically useless: old emails, instant messages, mobile phone positioning information, records of transactions from years ago, radio frequency messages recording small events, records of fleeting visits to web sites, old blogs, wikis and web pages that have long been withdrawn from public view.

The problem for the CIO is not that all this information is required, or has a value: it is, rather, that they do not know – or it is difficult to know – which information may be required, perhaps for compliance, evidentiary or analytical purposes, and which may never be needed again.

The problem is further complicated by the fact that, while a “keep everything” approach is feasible if expensive, there are many laws that outlaw the unnecessary retention of personal information. Moreover, keeping everything is not the same as making everything available, at the right time, to the right person. That requires a policy-driven, architectural approach.

Fear, opportunity, overload

Fear is perhaps not the best reason to invest in a new technology, but it has helped to focus the minds of senior managers and freed up funding for information management. Even so, fear of data security breaches, or of non-compliance with a slew of laws relating to how information is managed, has given CIOs a persuasive reason for making new, strategic investments.

It is not surprising. In 2005, major US companies reporting information security or management problems included CardSystems Solutions, which serves many major credit card companies, Bank of America, ChoicePoint, Citigroup, Morgan Stanley, Time Warner, IBM, Ameritrade, LexisNexis and MCI.

In the case of ChoicePoint, a $15 million fine from the Federal Trade Commission followed its failure to control exactly who could get access to its customer information.

Emails are a particular problem. Morgan Stanley was fined twice for failing to find emails relating to particular SEC investigations. And in 2004, UK-based financial services provider Bradford & Bingley was fined £650,000 by the Financial Services Authority for failing to present documentation relating to the mis-selling of financial services – it was forced to pay a further £6 million in compensation to almost 7,000 of its customers who were affected.

In many of these cases, a strategic review and an information policy would have reduced or avoided major problems.

In banking, a major concern is the Basel II Accord, which allows banks to risk lending more of their capital in exchange for conforming with more stringent risk management rules. This covers IT systems and the reliability and use of information. A more stringent Basel III Accord is currently being discussed.

This is forcing financial institutions to put in place effective and stringent information management policies. An example is the Nationwide Building Society in the UK.

“We are collecting data from the systems used in risk-based products, such as mortgages, loans and credit cards. We are marshalling the feeds to ensure the data is cohesive, and then loading the data into the data warehouse. We will also have reporting and monitoring tools in place to show that we are accurately managing risk, “ says Mike Humphreys, head of technology development.

A second, more positive reason to invest is that effective information management can bring dramatic results – especially where unstructured and structured information is brought together, for example, at the point where a customer might be making a decision.

An example: Pharmaceutical company Schering has brought together information from its Siebel CRM system, its Lotus Notes collaboration and document management system, and a number of other sources, so that its reps can close and sign off a deal when they are out on site visits. Their portal application – in this case supplied by BEA’s Plumtree unit – links structured and unstructured information, along with structured applications, into one single view.

Another example of effective information management can be seen at Capital One, the credit card company. Its use of advanced business intelligence techniques, including so-called genetic algorithm technology, is legendary.

Its process was described by a spokesman to Information Age: “At Capital One we have deployed an information-based strategy, which is based around getting customer data from whatever sources are available, loading it onto our systems and analysing the hell out of it.” The insight this process brings has enabled the company to rapidly introduce products that rivals have not been willing to risk offering.

Chris Slater, director of information resources for the business information company Experian, also believes in the importance of “using the moment”, arguing that customers increasingly choose the moment when they interact with a business. At that point, companies should use all the information they can muster in order to “seize the moment” and provide the customer with a powerful offer at the right time. “It is my fundamental belief that if your company can personalise its offering in real-time when the customer chooses to interact, you will fulfil the consumer-to-business (C2B) challenges that lie ahead,” he says.

But even if these new opportunities did not present themselves, and there were fewer compliance concerns, there is yet another case for information management: overload.

Various and numerous research reports show that employees spend at least one hour of every day on average looking for information, that email and instant messaging is destroying attention spans and productivity, and that most of the information that a business holds, even in digital form, cannot be easily accessed by the people who need it. And information volumes are rising rapidly.

Effective information management will enable CIOs to take an intelligent, considered approach to reducing these problems.

Information Management: the expert view

Clive Holtham is Professor of Information Management at the CASS Business School, London.

From the very beginnings of the computer industry, from 1951 onwards, the emphasis on information management – which computers are able to do very well – was hijacked by technology and technologists.

But after the dot-com crash, companies could never again play the technology card in the same way. So while information management is intrinsically important to IT, the shift of emphasis to the information side of IT is an opportunistic reaction to the technology aspect being less open to discussion.

In terms of structured and unstructured data, I worry whether these are converging or not. We are seeing a very commercialised form of convergence which is simply about the part of the unstructured data that can be converted into a technology-based format.

There is a ‘Google effect’ sweeping the enterprise. Before the days of search, we created metadata and structure in our information. And whether we call it indexing, tagging or whatever, creating that metadata forces us to consider how we structure information and how we sort it into fields. I think search is a way of avoiding those difficult decisions about how to have some structure in unstructured or semi-structured data.

We need to have a clearer view in our own minds about what information management entails, and we need to work with our colleagues to ensure the whole organisations is better clued into information and knowledge. Sadly, at the moment, some of the front-line managers are more interested in the technology than in the knowledge that the technology is there to support.