Software testing is universally acknowledged as an essential part of the development process. And for most organisations that build their own software, it is an integrated component of that process.

Research by IT analyst company Pierre Audoin Consultants has found that in western Europe, 80% of expenditure on software testing (€18 billion) is spent internally. Most businesses that write their own software use their own staff to test it.

Of the 20% of software testing expenditure that is outsourced (€4.5 billion), two thirds goes to the external development partner that built the software. That means more than 90% of software testing (by monetary value) is done by the same company that wrote the code.

For many years, companies that provide software testing services have argued that this is strategically unsound; that a company that both develops and tests the same software is “marking its own homework”.

As PAC’s numbers demonstrate, however, their protests have gone largely unheard. Just 6% (€1.4 billion) of software testing expenditure goes to a separate organisation from the original developer.

But that, according to PAC’s UK research director Nick Hayes, may be changing.

“For a lot of companies, testing is buried within development, and there’s no visibility into how much it’s costing,” he says. “But over the last two or three years, we’ve seen organisations say, ‘There must be a smarter way of doing this.’”

Having been forced to achieve visibility into testing costs, he adds, businesses are more inclined and in a better position to outsource their software testing, or to split it out from their outsourced development contracts.

Meanwhile, Hayes explains, the same focus on cost is prompting services providers to offer new and innovative ways to test software, as customers grow wary of the time-and-materials charging model.

The independence argument

Ask an independent testing services provider, and they will argue that the failure to divide development from testing has been responsible for some of the most grievous IT project failures.

“Why is it that for governments and big businesses, the bigger the system is, the more likely it is that it won’t work?” says Rudolf van Megen, CEO of German testing services provider Software Quality Systems (SQS). “The answer is simple: if you are an IT services company doing both testing and development and the project runs behind schedule, you can just cut down on testing time!”

Van Megen argues that only by splitting testing off from development can the customer be sure it is given the priority it deserves. This is only getting more important, he adds, as software systems grow both in complexity and criticality to business.

IT services companies like Logica, which increasingly sells testing as part of a wider development and delivery contract, rebuke this argument.

“A lot of the testing work we do is in projects where we both build and support a system,” explains Brendan Byrne, practice lead for Logica UK’s test and transformation division. “For those projects, we have to make sure that we have done the testing properly because we have service levels that we’re expected to meet during delivery. But we use the same testing resources and processes, regardless of whether we are supporting the system or handing it over to the customer.”

Continued…

Page 2 of 3

For ‘pure play’ providers like SQS and Indian testing services company AppLabs, which sell testing and testing alone, getting a separate all-purpose IT services provider in to do the testing is not enough, as they will inevitably pitch for the development business.

“In a typical project, 70% of the budget will be for development and 30% will be for testing,” says AppLabs CEO Sashi Reddi. “So the guy who won the testing business, it becomes his job to win the development part of the business as well, where the real money is.”

This motivation can affect the service they provide, he claims. “One of our financial services customers previously hired one of the top IT services companies for testing because the CIO believed in separating it from development, which is great,” Reddi claims. “But within a few months, he found that the testing provider was saying, ‘The quality of the code is so poor, if only you had given the development work to us you wouldn’t have been suffering like you
are now.’”

Again, though, conventional IT services providers have counter arguments. Logica’s Byrne says that the fact that it supports all stages of the development process means that its testing staff have a broader range of experience. “They will have worked in requirements, and they will have worked in support, so they understand the whole development lifecycle and understand the process upstream and downstream from them.”

“We wouldn’t use testing as a Trojan horse,” he adds.

Mahesh Venkataraman, who manages the testing innovation centre at Indian outsourcing giant Wipro, uses almost the opposite argument to dismiss the suggestion that commercial interests might compromise the objectivity of its testing. “Structurally and culturally, there is insulation between testers and developers, so there is no conflict of interest,” he says.

However, his colleague Sanjay Seth, marketing and alliances manager for Wipro’s testing services, was less reassuring. “When you get a new account, you have to mine the account,” he said in the same interview. “That is our sales strategy.”

For PAC’s Hayes, there is clearly something to be said for keeping testing independent from development. “If you look back at some of the UK public sector transformational IT projects, would they have benefited from an external provider coming in and managing the test functions?” he asks. “They probably would have done.”

However, managing the testing process across suppliers raises its own issues, he adds. “When you divorce testing from development, you need to make sure that the two sides work in a collaborative way,” Hayes says. “For example, you need to make sure the testing company’s project deadlines are in sync with the development company’s deadlines.”

Continued…

Page 3 of 3

Hayes says that the recession drove many businesses to get a handle on precisely how much they spend on testing, and in what manner. In many cases, that raised questions about whether the conventional time-and-materials business model that most testing services providers employ serves their best interests.

“The economic situation prompted a lot of businesses to have a good look at testing,” he says. “Once they’d split it out, they started to look at whether there might be a smarter way to procure testing, in order to produce the right sort of behaviour from suppliers.”

The suppliers, for their part, are experimenting with new ways to deliver testing services.

SQS, for example, now offers ‘managed testing services’, where customers commit to multi-year partnership contracts. This allows SQS to get to know its customers’ requirements better over time, the company says, and gives the customer reassurance that it will have the testing expertise it needs over time.

“Customers want to know they can depend on these [testing] resources,” comments van Megen. “We sign an service level agreement that means we have to guarantee certain resources will be available to them.”

He claims that a managed services approach helps to relieve the burden of testing systems after routine maintenance or upgrades. “A lot of the testing after maintenance is repetitive work: you might only make a small change but you have to do lots of retests,” he explains. “This is a huge effort, and it is normally spread between IT and the business departments.”

Wipro, meanwhile, is moving in the other direction with its recently launched ‘testing-as-a-service’ offering. This allows customers to procure testing resources through an online portal.

“The conventional model for procuring testing services is for the customer to provide a request-for-proposal, then a bunch of sales guys call them up, a lot of presentations go back and forth, and there’ll be negotiations and finally a contract,” explains Sanjay Seth. “This entire process can take anything from 90 to 270 days. That’s fine if you are doing a multi-year engagement and it is a transformational deal. But there are many kinds of testing that can be done in days or weeks, so you don’t want to go through a nine-month sales cycle.”

The company hopes this service will allow it to capitalise on the untapped market for small, ad-hoc testing projects, such as performance testing on a website. “We see this as a huge opportunity to expand our market,” says Seth. “In three years’ time, we hope that 15% to 20% of our testing business will be sourced through the portal.”

A related testing services innovation is AppCloud, a cloud-based performance testing service from AppLabs. The service is essentially an implementation of open source performance testing tool JMeter on Amazon’s EC2 cloud computing platform. This makes it possible for customers to pay for performance tests as and when they are needed.

AppLabs’s other approach to differentiating its testing services is to develop its own intellectual property. In the past that has meant customising open source testing tools for the requirements of enterprise customers, but more recently it has developed its own proprietary testing management processes.

One recent example is what the company calls its Enterprise Test Automation Platform, or ETAP, a framework that helps its customers plan how they use automated software testing tools.

“When a company wants to automate software testing, they will typically go out and buy a tool like [Hewlett-Packard’s] QTP,” explains Sashi Reddi. “But there’s rarely a centralised strategy for testing automation.”

“ETAP is a framework that we developed for ourselves,” he explains, “but we now offer it to customers to help them work out the return on investment for automated testing, how much of their testing should be automated, and what tools they should use.”

Intellectual property like ETAP is helping AppLabs win business against the larger IT services suppliers, says Reddi. Even so, he reflects, the independent software testing services vendors command only a fraction of the overall testing market.

For that reason, Reddi hopes that rival independent testing services suppliers, like SQS, succeed. If there were more viable ‘pure play’ providers, he says, more companies would consider using their services. “It would be to our advantage to have another five to ten companies like us, because then the market would shift to the pure plays.”

As it is, independent pure play testing providers share a tiny corner of the market. “It’s a huge market, but its all being split up by the big guys,” Reddi says. “We only get the crumbs off the table.”