Information Age: News, analysis & insight for IT & business leaders

By combining complex event processing technology with sensors and tags, companies are finding ways to apply the analytical power of information technology to physical objects

Many software vendors, especially in the business intelligence space, tout their products as ‘real-time’ systems.

On closer questioning, however, many will concede that what these products truly offer is not ‘real-time’ in its truest sense, but ‘near real-time’. Some of today’s business intelligence (BI) tools, for example, can give a company the ability to analyse data and to extract meaningful business information faster than ever before – but only once that data has been consolidated in a database and then interrogated using querying, reporting or other analytic tools.

It’s a question of latency. Traditional BI processes, where data must be collected and processed, introduce a time delay that doesn’t matter much when it comes to end-of-month, end-of-week or even end-of-day reports. And these days, it’s fair to say that modern BI tools can, when engineered correctly, do a pretty good job at closing the window of latency still further.

But it is also fair to say that those tools are simply not engineered to handle continuous, streaming data such as live trading information from financial markets. In order to tackle that kind of challenge, companies in the financial services industry, for example, have increasingly turned to another technology – complex event processing (CEP) platforms – that eliminates the latency and restrictions associated with traditional BI.

In a sense, a CEP platform is like an inverse database. Instead of issuing queries over data, a continuous stream of data, or ‘events’, is passed over the query. Or, in the words of Mark Palmer, CEO of StreamBase Systems, a rising star in the CEP space, these systems allow companies to “find the needle before it goes into the haystack”.

Based on the results of this analysis, a CEP platform can then “prompt applications or people to react to information by resetting processing priorities, changing online sales strategies, buying and selling stocks, or some other action,” explains Mike Gualtieri, an analyst with IT market research firm Forrester Research. “Analysing historical databases cannot provide answers to these questions – they require the business to act on urgent events as they happen. CEP platforms connect to live events from wherever they originate, be it a message queue, a click stream or a market data stream.”

Beyond banking

To date, however, CEP has struggled to find a strong foothold outside the financial services niche – but that is changing. Other industries are increasingly interested in capturing real-time events, but in many cases the events in question are not the system-to-system messages generated by trading applications but the alerts issued by sensors and tags attached to physical objects moving about the real world.

That will open up whole new areas of market opportunity for companies offering CEP platforms, which include IBM, Oracle, Progress, Sybase, Tibco and, most recently, Microsoft, according to Teresa Jones, an analyst with IT market research company Gartner. In September 2009 she estimated that market penetration stood at somewhere between 1% and 5%.

“Event processing is underused partly because, until recently, relatively little data on current business events has been available in digital form,” she says. “In the past, many events were either undetected or detected but not reported in a digital format that could be sent over a network or manipulated by computer.”

But the amount of available event data has escalated rapidly in recent years, she adds, and now amounts to an “explosion of event streams flowing over corporate networks”.

Continued...