It is difficult, at a high-level conference dedicated to the impact of search technologies in the enterprise, not to run into evangelists on the subject, whether they are on stage, in the elevator, or live blogging from the hotel room next door.

But no-one captured the sense of excitement around the technology as much as the effervescent Chris Weitz, manager director and search solutions practice leader at BearingPoint, the management and technology consultancy: “This is the dawn of search in the enterprise. It will not look like the Internet…it will be better than the Internet.”

This message, he adds, is only just beginning to sink in. “Everyone who writes code for a living is now declaring that search is their differentiator.”

Weitz gives the new Microsoft Vista operating system as an example: “Vista might as well be called the Search OS. You can’t avoid it. It’s infected with search.”

The enthusiasm for search is understandable. At the FASTforward conference in February 2007, search technology was variously presented as a near-universal interface, as a powerful new addition to the armoury of business intelligence tools, as a form of middleware, and even as a possible alternative to the relational database.

Indeed, John Marcus Lervik, the CEO of FAST, believes that search is becoming “a platform” – integrating and providing ready access to information buried in applications, in underlying search engine ‘silos’, and in structured databases. “You need a mechanism for bringing it all together, a kind of intelligent glue. In that sense, search is becoming a platform,” he told the San Diego audience.

A key debate in the coming years will be to what extent search tools can or should complement traditional databases and BI tools (see article, Expanded intelligence), and to what extent they should replace them.

Lervik believes that search engines, with their flat, fast and flexible indexes, should replace databases in situations where “databases shouldn’t be used”. Database technology, he said, “is not very good at talking to you, at presenting the metadata to you. You need to know exactly how to retrieve it. People don’t know that.”

And second, the metadata cannot easily be changed. That not only limits business agility, but means that structured databases on their own have a limited role to play in the new social networking world of Web 2.0 or Enterprise 2.0. End users want to add, change and tag data, with ease and at will. A third problem: relational databases are simply not fast enough – search indexes can be created rapidly and are extremely fast at retrieval.

Many organisations are now integrating search and structured databases. At FASTForward, three big companies – Reuters, ING Bank and Merrill Lynch – all described applications which use search technology as a way of rapidly extracting data from underlying structured databases.

Ray Tomkins, search engine advocate for Reuters, described how its financial information service is being redesigned. A simple search button has been added, enabling users to get to information they need without needing to know any predetermined structured codes. Eventually, all the services on the screens “will work in a simple, user-friendly, Google-like way,” said Tomkins.

Replacing relational databases may be a step too far for these and other companies. “There’s a lot of conservatism in this area. But search can be the middleware that links things up,” says Bjorn Olstad, FAST’s CTO.

Part of the conservatism stems from businesses’ experience with Internet search engines, which tend to be imprecise and ambiguous. But enterprise search can be fundamentally different from searching the Internet, and so can be implemented in a way that is much more precise.

First, users can apply taxonomies and ontologies, and this can even be done implicitly and after the fact. In other words, the organisation can apply a layer of structuring metadata to overcome ambiguities and apply context.

Second, there is a big opportunity for the system to understand the context of the search (see box). And third, search engine suppliers can use techniques such as linguistic analysis or entity extraction to ‘clean the data’. For example, a purchase order number appearing in an email can be extracted and checked against the transactional system, thus helping to identify the subject of the email.

For all his enthusiasm, Chris Weitz warns enterprise users not to underestimate the difficulty of liberating the “long tails” of information from corporate databases and from legacy applications. “It’s very early for search in the enterprise because the problem is infinitely more complex. It’s more complicated than on the Internet. There are no magic bullets.”

Further reading in Information Age

The invisible interface