Over the past decade, online auction company eBay has played a central role in the world of ecommerce. By some estimates, as many as a quarter of online users have visited its virtual trading site. And yet, as the company continues to grow, its technology executives have come up against a tough problem: they simply cannot buy the technology that they need.
In the 12 years since the company was founded in a Silicon Valley living room, it has built a vast technology infrastructure. But unlike that other Internet behemoth Google, eBay’s leaders are less fanatical about engineering the technology that powers their business.
As Paul Strong, a distinguished research scientist at eBay explains, being technological groundbreakers is a by-product of the business, not its core aim. “Our business is to provide a market for people to buy and sell online. We never intended to push back the boundaries of technology; it’s not what we’re there for.”
Today, eBay has an eye-watering array of technology that underpins its global websites. Its 15,000-strong pool of servers run over 600 production databases and 2,000 instances of search technology; it processes 26 billion SQL transactions each day as part of the 100 million online auctions that take place.
Managing such complexity has been a huge challenge, not least because the transaction times experienced by the users have a direct impact on eBay’s bottom line. This large distributed infrastructure simply has to run as close to optimum performance as possible – every second of every day.
But the challenge is made trickier because off-the-shelf management tools cannot cope with the scale and complexity of eBay’s distributed, heterogeneous IT estate. “There are packages that do certain bits very well, but there is nothing that can manage everything. So we’ve had to become experts at building our own tools,” says Strong.
A good example of the problem eBay faces is server virtualisation technology, which is lauded in many quarters for providing an efficient method of managing workloads across multiple machines. But this extra layer of technology can result in a performance “hit”, notes Strong, and when your services are zealously attuned to latency, that can be problematic. “If it adds 10 milliseconds onto transaction times, it starts to hurt.”
So while virtualisation remains a “fantastic technology for some circumstances”, it cannot provide the whole answer for eBay. Instead, Strong needs management tools that can monitor the entire distributed technology stack, from application to network and storage. This is what he regards as grid computing – although he readily concedes that definitions of ‘grid’ vary enormously.
Strong hopes that eBay, through its involvement with the Open Grid Forum, will be able to encourage the development of a standard set of tools for managing such sophisticated distributed environments, irrespective of naming conventions. And if vendors can use the OGF to ensure their management tools work across complex heterogeneous environments, Strong thinks he could put them in touch with at least one potential buyer.
