In 1997, when John Lervik and Espen Brodin set out to reinvent the world of search engines, the co-founders of Fast Search & Transfer (Fast) made a serious miscalculation. “We thought it was a technology game,” says Silvija Seres, Fast’s VP of strategic market development told the Business Intelligence 2006 delegates. “Google realised it was a media game.”

Google is now a $100 billion company that is as synonymous with Internet search technology as Microsoft is with PC software, and one that last year grew revenue 92% to more than $7 billion. Meanwhile, Fast’s revenue in its last fiscal year was a more modest $103 million. However, even if it meant losing out in the race to win the hearts and minds of the world’s web surfers, Fast’s decision to concentrate on developing first class search technology may still play dividends – in the emerging market for enterprise search engines.

Investment in search technology still trails behind that in business intelligence (BI) and data warehouse technology at most corporations, but that may not always be the case, says Serres. Although search is still regarded as primarily a needle-hunting utility, useful for sifting odd nuggets of information from the haystacks of unstructured data that litter the corporate software estate, pioneering corporate users of the technology are constantly discovering new applications for it.

Among Fast’s 3,500 enterprise search customers, Seres is able to point to a variety of different applications. At IBM, Dell and Vodafone, for instance, Fast search technology is deployed to give consumers faster and more accurate access to free information, whereas at AutoTrader.com and Sensis, Fast provides the same efficient access to paid-for products. At GE, DaimlerChrysler and Goldman Sachs the emphasis is on providing employees with better access to their company’s own information.

All of the above are examples of search technology deployed against its traditional target – unstructured data. But, there is no law that says search cannot benefit owners of structured data; at Lexis-Nexis, EMC and Reuters, Fast is providing users with more flexible access to contents of SQL databases.

It is these latter applications which have the potential to create the biggest stir among corporate information managers. With data volumes growing by 200% a year, and 66% of workers now spending around 30% their time of trying to extract useful information from these swelling data silos, conventional approaches to information discovery and access based on structured data are running out of steam. “Traditional applications grow to be too complex and too expensive as queries become more sophisticated,” said Serres.

Information underload

Queries, too, are becoming more sophisticated. In 2003, the average word length of queries submitted to Google was a little more than one. Today, most Google queries are submitted in three words or more. This, says Seres, is creating a perception among knowledge workers of a “sea of information” which is easily navigable, and open to interrogation in an ad hoc fashion. This is a far cry from today’s structured data warehouse and BI-based regimes, where accurate answers require prior knowledge of the question.

Using search technology, claims Serres, Fast customers are discovering that they can dispense with much of the cost and complexity of building structured BI systems. “They no longer need to know what users want before they ask,” she said.

These are big promises, and delegates at the BI 2006 conference were clearly sceptical. As one delegate summarised, “No data warehouse, no ETL, no idea.”

However, Serres is not actually promising an instant revolution. At the moment, Fast’s search technology is being positioned as a platform to form the basis of new access tools that will complement, rather than replace existing BI investments.

In this auxiliary role, search offers conventional structured systems a new way of dealing with “fuzziness” and a better shot at supporting traditional SQL responses with the context that can make raw data more meaningful. It may also reduce some of the back-end costs associated with data cleansing and grooming, which can be such an obstacle to querying of data originated across multiple application silos.

Today, enterprise search is in its infancy, and like so many emerging technologies will probably deliver less than many of its proponents currently promise. However, whereas today, says Serres, “people talk about information overload, what we may have is really information ‘underload’.”

Further reading

• Search 2.0 - lead feature on enterprise search, March 2006
• The evolution of search - new technologies reviewed, March 2006
• When worlds collide - unstructured information converes with structured application and transactional data, February 2006

More on search in the Information Management Briefing Room