“Why is everyone getting so excited? We had ESB [enterprise service bus] technology 15 years ago,” said Ram Menon, the chief marketing strategist for Tibco, the integration software company, during a recent conference call.

Menon is a little sensitive on the subject. At every turn, he finds analysts, customers and the press are talking about the disruptive, low-cost technology of the

ESB, which proposes to replace complex integration hubs with distributed, light-weight, standard-based systems.

“It is hype,” says Menon, comparing it with dot-com era technologies such as B2B exchanges. The ESB, he says, is just a low-cost connectivity technology that has so far had little significant impact on the $2 billion annual market for enterprise integration products where Tibco is the market leader.

“In every case, we’ve seen people who want to buy an ESB take a whole Tibco product,” he says, dismissing the popular argument that ESB products can replace integration brokers – such as Tibco’s – that typically sell for over five times the price.

Ross Altman, the chief technology officer of SeeBeyond, another leading supplier of enterprise application integration (EAI) technology, agrees: ESB technology, he says, does not meet the specialised requirements of most enterprise customers, and won’t for many years. And if functions are added to the ESB products to meet those requirements, then the disruptive, price advantage of the technology is lost.

Now the confusing part: although they don’t talk about it much, SeeBeyond and Tibco both have products that are described, on their web sites, as enterprise service buses. And, moreover, when Gartner analyst Yefim Natis listed ESB leaders at the European Application Integration and Web Services conference in Amsterdam in June, he put Tibco and SeeBeyond firmly on his list.

That surprises Gordon Van Huizen, the CTO of Sonic Software, the Progress Software subsidiary that is usually credited with inventing the lightweight ESB technology. He says that most analysts view ESBs as an important, new category of product with a distinct, new architecture. And it is not an architecture shared by older EAI products, he insists.

In fact, some observers, among them many leading analysts, take a fairly radical stance on the emergence of the ESB, likening its role in integration to the emergence of the relational database in software or the IP router in networking. Eventually, they think that almost all integration will use ESB, rather than traditional ‘hub-and-spoke’ integration brokers – just as almost no one today uses hierarchical databases or non-IP routers.

The rapid emergence and success of the ESB, says Steve Craggs of Saint Consulting and European vice chairman of the Integration Consortium, explains some of the confusion. “The marketers are deliberately pulling the definitions in all sorts of ways,” he says. Those established vendors with older products, he says, “will attempt to create a marketing disruption. They will say ESB is a good idea, but that ‘we already have one’.”

And in most cases, these vendors either do have an ESB, or they will have one soon (see table). But the question is: What is it that they will have? And why are so many suppliers keen to have an ESB, yet simultaneously to play down its disruptive significance?

The business problem

The answers can be found by looking in two directions – at the immediate and often troubled past in EAI, and at the more promising ‘service-oriented’ future.

Effective data and business process integration is critical to the modern business. In almost every field, organisations are attempting to build end-to-end automated processes that reduce or eliminate expensive, time consuming and error-prone human intervention. They need to support those often-complex sets of processes with applications. The problem: rarely does a single application – or even an applications suite – support the full range of processes. Moreover, the operational data used by different applications is typically structured and stored in inconsistent formats.

EAI brokers or integration hubs, which connect the applications that support a process and orchestrate the transformation and the movement data between the applications, have been the answer. Over the past 15 years, integration software has become a $2 billion a year market, with related services accounting for much more.

But for customers, it has been expensive and difficult. Even today, most EAI hubs cost upwards of $300,000 and typically, systems development work can cost five times that. Such a price tag means that EAI has been largely confined to big customers with pressing problems.

Furthermore, studies have shown that many EAI projects fail to meet all their objectives. Projects often become mired in complexity, and many planned integration links are never implemented.

Lightweight revolt

This is where the ESB finds the first of its two customer groups – simplifying integration.

The ESB discards, or at least plays down, the importance of the central, proprietary hub, and replaces it with intelligent software that is installed at every participating node. This software is capable of reading the content of a message and routing it accordingly – there is no need for a message to be sent via the hub.

A second innovation: transformation of data, or business process management functions (such as workflow and orches-tration) can be added to the service bus just like any other application service. If transformation is necessary before another application can understand it, then the message will be routed to a transformation engine. If not, it won’t be.

Using transformation and XML, existing applications can be replaced or upgraded with minimal disruption. “With an ESB in place, operational IT flows can move seamlessly between different components, departments, disciplines and other companies,” Craggs explains.

This new architectural approach has had a disruptive effect on the integration market, not least because it is much cheaper. Customers do no need to buy a whole stack of functions if they don’t need them, and can implement tactically. It is estimated that about 80% of all messages, for example, need no transformation. So why buy and install the software?

From a standing start in 2001, the ESB market today is worth around $80 million, with Sonic the runaway early leader, with sales of around $30 million. Overall dollar licence revenues of non-ESB EAI products, meanwhile, have been largely falling or static during this period. Lightweight ESB sales are expected to triple within three years.

This has some analysts excited. IDC, for example, has described the ESB as a “disruptive technology that will extend EAI capabilities, once only available to large, extremely well-funded and heavily-staffed IT shops” to a wider user base.

Most of the 200 odd implementations of ESBs around the world today have focused on this tactical area of reducing integration costs. ABNA, part of British Associated Foods, for example, uses Sonic in this way. ABNA’s ESB, says Mysia Benford, the company’s logistics and IT director, provides “a beautifully isolated messaging platform at far less cost than a point-to-point or hub-and-spoke integration approach.”

However, it is also in this area that ESBs are most controversial – because, says Ross Altman of SeeBeyond, “lightweight” was really a code word for “cheap”. And this, he says, is no longer as true as it was. “The definition (of ESB) has morphed,” he says. Now, ESBs increasingly include many of the functions found in EAI engines – such as transformation, adaptors and BPM. And, he adds: they don’t necessarily provide the control, resilience or security of traditional EAI hubs.

“The whole notion that ESB messaging is cheaper doesn’t apply any more,” Altman says. A $40,000 product can quickly become a $100,000 implementation once all the extras are added in.

This is hotly disputed, of course, by ESB specialists such as Sonic and SpiritSoft. They say licence and implementation costs are still far apart. Most analysts agree that the gap has not yet been closed – but also say that traditional EAI hubs are best used when requirements are more complex.

Craggs of the Integration Consortium makes two critical points. First, ESB functions can be bought and added on incrementally as needed – so it is much cheaper for those who have simple requirements; and second, ESB products have dragged down the prices of traditional EAI products, narrowing the price gap.

But the real promise of the ESB lies ahead, with the emergence of the SOA (service-oriented architecture). Standardisation and web services mean that ultimately, all applications and business processes – internal and external – will be accessed, integrated and combined as plug-and-play web services, all part of the so-called SOA. No one is under any illusion that this will be easy or cheap; but ultimately, it will dramatically increase business flexibility.

This is where the ESB will play a critical role: “The middleware infrastructure for the SOA will be the enterprise service bus,” says Gartner analyst Massimo Pezzini. Increasingly, he adds, even the big software vendors, offering application suites will base their products on an ESB foundation.

This makes obvious architectural sense. If every other business function is delivered as a service on a bus, then why not the integration function too?

This ESB foundation, however, will not necessarily be particularly lightweight; Gartner, in fact, is using a general archi- tectural definition, in which standards, such as web services, are used wherever possible, and where integration functions are distributed around using some form of bus.

The Future ESB

This wider definition of the ESB has made vendor differentiation difficult, blurring the lines between several types of products. Even the notion of a core bus architecture is not essential, since it is possible to build a bus by stringing many hubs together, or delivered integration services from a hub as if they were distributed services.

Moreover, some ESB vendors don’t even supply the transport mechanism, but orchestrate and integrate on a layer above the messaging level. These vendors – companies such as Polarlake, Grand Central, or Blue Titan – are perhaps better positioned as web services management suppliers.

Clearly, however, all the big suppliers of EAI products are already developing their core products so that they include ESB functions – in the wider sense. This means that, if they do not do so already, these products will act as message-oriented, standards-based buses that integrate and orchestrate applications, data and services.

They might not be able to tick all the boxes today, but IBM, Tibco, SeeBeyond, WebMethods, SAP, BEA and Microsoft will clearly be positioned in this way. Some heavy re-engineering work is likely to be needed to make some of these proprietary hubs look and act like standards bases buses.

But it will be done – over time. Indeed, there is still an opportunity for market leadership – and for new players. “There is an opportunity for newcomers to provide a deep and powerful ESB,” says Pezzini.

With the ESB playing a pivotal role, and with most organisations moving towards the SOA, this will drive huge growth. Gartner expects that the majority of large organisations will own an ESB by end of 2005 and 10% of all new application integration will be ESB based by 2006. Beyond that, it will grow further as the SOA takes hold.

But what of the lightweight sector? Will customers still be able to pay less? The impact that Sonic and others have already had is irreversible. Most of the major suppliers are now adopting a two-speed approach to the ESB, planning separate or stripped down versions of their products for those customers who want or can only afford simpler solutions.

To speed up the process, says Pezzini, “the pure play integration companies will make defensive acquisitions in the ESB area.”

Even Sonic is expecting its market to be mainstream shortly. Its strategy, says CTO Gordon Van Huizen, is to exploit its technology lead and establish a strong position before all the rivals crowd in over the next two years.

ESB and SOA will be central themes at the Information Age XML &Web Services conference in London on 29 &30 September. Details at www.xmlwebservices.co.uk.