Information Age: News, analysis & insight for IT & business leaders

Enterprise search is growing fast, but it has yet to prove whether it is an add-on feature of content management or a technology category in its own right

Driven by compliance obligation and the sheer volume of information pouring into their organisations, the amount of data companies are storing is estimated to double each year. For the average employee, navigating relevant parts of that information mountain is a formidable and wasteful exercise.

But in the past few years, enterprise search – which applies some of the same principles as Internet search to unstructured business information – has promised huge productivity gains. “The majority of employees do not either have the time nor often the skills or training to search efficiently and effectively,” says Ovum analyst Mike Davis. “Automated enterprise information access is a major opportunity for improving productivity, reducing error and corporate risk.”

Legal discovery is certainly a major driver for the sector, but anecdotally a surprisingly large number of enterprise search deployments are user driven – typically an influential user asks the question: “Why can’t I search my firm’s files as easily as Google searches the Net?”

Users become frustrated when they go home and discover the tools they use to surf the web are more sophisticated and efficient than those they are forced to use at work, says Google enterprise’s product marketing manager for EMEA, David Armstrong.

Google’s plug-and-play enterprise search appliances are used by companies such as Apple and British Airways for their public-facing websites. However, the web remains a very different beast to the enterprise environment, and some question whether the search giant has yet made the leap.

Unlike the web, which is fairly consistently constructed of HTML pages that want to be found, unstructured data in an enterprise includes emails, documents, files and even third-party data strewn across a multitude of systems, some of which the enterprise may even be unaware it owns.

“Google has an incredible brand, but the reason it hasn’t been as successful in enterprise search is because it is a very different field [to the web],” says Autonomy CEO Mike Lynch.

“Unlike the web, [enterprise] information is spread across multiple repositories and perhaps 300 to 400 different file types and formats. Search algorithms on the Net work by looking through the most popular pages – but the important legal document you’re searching for, for example, might be unpopular. If you miss it, you [could] go to jail.”

The Butler Group calculates that 80% of the data stored in an enterprise is made up of this unstructured data, as opposed to structured data contained in a database. But that does not always align with perception: an HP study found the average European CIO thinks only 25% of their data is unstructured. One of the key drivers for enterprise search is legal discovery, because all this unaccounted ‘lost’ data (which HP cheerfully terms “information dark matter”) represents a massive compliance risk, if not an operational one.

The last few years have seen a booming number of search projects; Gartner predicts the market for enterprise search will grow 15% this year, topping $1.2 billion by 2010, with the technology pervasive by 2012.

Information Age readers are ahead of that curve, according to our Effective IT survey. Almost a quarter of respondents are planning to implement the technology in the next 12 months, while a further quarter have already deployed a corporate-wide search platform, most for several years.

Of those who have an implementation, over half describe it as “effective” or “very effective”, with the remainder neutral. Only a tiny percentage report their enterprise search capability as ineffective.

However, the search technologies available are far from equal. Paul Ellerbeck, IT director at online listings site Fish4, advises seeing multiple proofs of concept, preferably side by side. “It’s very easy to be sucked into the belief that a search technology can do what you want it to do,” he cautions.