It is easy to pay lip service to the power and value of information, and most organisations do. But instilling a leadership culture that pays heed to facts and analysis is far harder.

According to a survey of 257 UK businesses conducted by management and technology consultancy Accenture, 40% of decisions are still made by intuition alone. But Astrid Bohé, executive director at Accenture’s information management practice, argues that intuition is often wrong.

“We once worked with a company that wanted to understand the effectiveness of its different retail channels,” Bohé recounts. ”They had a preconception about one particular retail channel that was very expensive and they expected to use it less in future. It felt like we were brought in to support what they already thought.

“But we found that the most expensive channel produced the most profitable, most loyal and longest-tenured customers,” she says. “Based on this analysis, our client changed their distribution and marketing practices. They were able to significantly optimise acquisition costs because they were acquiring the right kind of customer, even if it was more expensive to acquire each one.”

"There is no correlation between what companies spend on BI and their degree of success"

The recession, Bohé adds, makes relying on intuition all the more dangerous. “When money is tight, you need to make fact-based decisions,” she says. “It is too costly to get things wrong.”

Perhaps it is this realisation that lies behind the recent recession-defying performance in the market for business intelligence (BI) software. In 2008, the global market for BI platforms, analytic applications and performance management software increased by 21.7% from $7.2 billion to $8.8 billion, according to analyst Gartner.

IBM has cited business analytics as a strategic area of focus in its last two financial quarters and has made significant investments in the field, both in the form of its Smart Analytics centres (see below) and the $1.2 billion acquisition of analytics software provider SPSS in July 2009. Bernie Spang, IBM’s director of data and Infosphere strategy, explains why BI and analytics are so important at this moment in time: “There is a perfect storm of the maturing of the technology, at a time when businesses themselves are taking a step forward in the maturity level of how they use their information, combined with the global economic situation.”

The flaws of business intelligence

This does not mean, however, that all is well in the world of BI. Most large organisations have attempted some form of BI deployment, and although there are plenty of success stories (see Case Studies box) there are also a great many dissatisfied customers.

“There has been a lot of spending on BI but, ironically, it hasn’t been done in a very intelligent way,” says James Richardson, a BI analyst for Gartner. “There is no correlation between what organisations spend on BI and their degree of success.”

Richardson has identified a number of common yet fatal flaws in BI deployments. Some of these are familiar bugbears in enterprise software projects – lack of business alignment, lack of strategy, taking whatever the platform partner has on offer without evaluating the competition – but others are peculiar to BI.

For example, poor data quality has earned BI a bad reputation in many organisations, he says. “What you can do is not let BI take the blame for this, but let BI be the window onto the data quality problem.”

Another problem for BI deployments can be the fact that by enforcing a so-called ‘single version of the truth’, you prevent employees from ‘massaging’ data the way they like to. This is a positive when it comes to producing accurate management information but it can be met with opposition. “The only way to resolve this is by having executive sponsorship saying that if anyone shows up with anything apart from centralised reports, they are going to take it as opinion, not fact,” says Richardson.

The sustained appetite for BI despite the above challenges and the dissatisfaction encountered by many adopters is testament to the value proposition of the technology. But it also suggests that BI and analytics is an area of enterprise IT that is ripe for innovation.

New approaches

Happily, that innovation is already under way. IBM, for example, has chosen analytics to be the subject of a new kind of offering that unites software, hardware and consultancy services. The company is in the process of establishing Smart Analytics centres across the world, each tailored to local industry. (London’s Smart Analytics centre, focused on the financial services industry, will be opened later this year.)

From these centres, IBM says, organisations can procure kit that is optimised for analytics software as well as the expertise required to establish an analytics function. This will dramatically reduce the time to deployment, says Spang.

“We can get a Smart Analytics deployment delivered to you in less than two weeks. Typically it would take months, plus you would need to hire all your own IT experts to optimise the equipment.”

Accenture’s Bohé says IBM is “going in the right direction” with its Smart Analytics centres, but she warns against relinquishing control of the BI project to a third party. “There still needs to be someone on the side of the business that understands where the value of BI comes from.”

Another direction of innovation for BI is, inevitably, onto the cloud. Open source BI software vendor Jaspersoft recently announced a partnership with three other suppliers (RightScale, Talend and Vertica) to provide scalable BI and data warehousing over the cloud.

There are two use cases that make ‘cloud BI’ applicable to the enterprise, says Jaspersoft CEO Brian Gentile. “Number one is the ability to do BI and data warehousing proofs of concept quickly and easily, to be able to quickly determine whether the BI tool will solve the problem without affecting the production environment,” he says. “The second use case is to build BI systems for temporary projects where ordinarily it would take longer to set up the BI infrastructure than the length of the project itself.”

But technological innovation is not only offering new ways to deliver BI and analytics; it is also changing the scale and speed at which complex data analysis can take place.

In order to process volumes of data never seen before, search engine giant Google developed its own software framework called MapReduce. The framework allows analysis workloads to be split between thousands of servers by chopping the workload into smaller jobs then recombining the results.

Well established among the Internet giants, MapReduce is now beginning to find business applications such as fraud detection in insurance.

“Companies are asking us how they can harness the power of MapReduce in their existing IT ecosystems without hiring a team of computer science PhDs, like Google and others have, to manage it,” says Tasso Argyros, CTO and co-founder of Aster Data, a company that integrates MapReduce into traditional SQL database environments.

The potential of MapReduce, and the related open source computing platform Hadoop, in enterprise applications is only beginning to be understood, but it promises to challenge the perception that there is a correct way to ‘do BI’. That is a perception that has perhaps prevented many organisations, who have unsuccessfully attempted the traditional model of BI, from achieving the analysis-based leadership style that the times require of them.