Information Age: News, analysis & insight for IT & business leaders

Enterprise applications are, by and large, very expensive. Bad news then that studies which compare enterprise applications on the basis of total cost of ownership (TCO) are deeply flawed, according to a recent report published by Enterprise Applications Consulting (EAC).

The problem, EAC found, is that the sheer number of variables that differ between implementations make comparisons invalid. TCO studies, which typically survey 200 projects, give no reliable indication of the relative cost of vendors' products over the entire life cycle, says EAC analyst Joshua Greenbaum. "TCO is a troubled measure with a troubled past," he says.

When the metric was introduced by Gartner analysts in 1987, rival firm Forrester dismissed TCO studies as 'hype'. Nevertheless, it soon became the standard cost measurement, with all leading analyst firms - including Forrester - now providing their own models.

The problem with comparing implementations for different customers is that no two companies are exactly alike. The functionality requirements vary between customers, as do the amount of existing in-house expertise, the cost of labour in the customer's part of the world, etc. Any cross-vendor TCO comparison that could support statistically significant conclusions would have to survey more customers than analysts could ever hope for.

Research conducted by enterprise resource planning vendor SAP found TCO evaluations by Gartner, Forrester and the Meta Group (now owned by Gartner) varied wildly in their estimation of average per-PC TCO for similar products (see graph).

So, should total cost of ownership measurements be ignored altogether? No, says Greenbaum, but analyst-conducted cross-vendor comparisons should be shunned in favour of the vendors' own TCO evaluation frameworks, such as one launched recently by SAP. Only when all vendors develop their own programmes for measuring TCO will valuable comparisons be possible.