‘Big data’ technologies Mapreduce and Hadoop are poised for rapid expansion over the next five years, according to research from IDC.
MapReduce is a technique for splitting large analytical jobs into smaller, distributed workloads. Hadoop is the most popular framework for building scalable analytics applications, using MapReduce and other techniques.
Together, Hadoop and MadReduce are "taking the software world by storm", said Carl Olofson, an information management researcher at IDC.
"[They are] inspiring a wide range of projects that collect both structured and unstructured data and produc[e] output that may be used to answer a single question," he said. "They serve as the foundation for a range of questions, queries, or searches; or [can] be loaded into a data warehouse for more systematic and repeatable query."
The market for software based on Hadoop and MapReduce was worth $77 million in 2011 – 0.25% of the total business analytics market – but IDC forecasts that it will grow by $147 million dollars per year on average until 2016, when the market will be worth $812 million.
The report found that the increasing availability of large, unstructured, databases from social networks and web applications is driving the growth in the Hadoop marketplace. The report cited interaction, sentiment and behaviour data as particularly important.
Despite the strong projected growth, IDC said that competition between open source vendors and closed source counterparts might inhibit growth, as having open source groups as such a large portion of the Hadoop ecosystem could inhibit revenue growth in closed source projects, pushing down closed source license fees.
"The Hadoop and MapReduce market will likely develop along the lines established by the development of the Linux ecosystem," said IDC research Dan Vesset. "Over the next decade, much of the revenue will be accrued by hardware, applications, and application development and deployment software vendors."
MapReduce was developed by Google as a way to process its vast volumes of clickstream and search data. Doug Cutting, an engineer at rival Yahoo!, built Hadoop as a framework to build MapReduce applications.
Last year, Yahoo! spun out its Hadoop-focussed division as a standalone company called HortonWorks. Today, HortonWorks, Yahoo! and big data start-up CloudEra are the top contributors to the Hadoop codebased.
Not all applications of MapReduce are based on Hadoop, however. Aster Data is a company that applies the technique to relational databases. It was acquired by data warehouse vendor Teradata last year.
