Data storage is not always the most exciting corner of the IT industry, but in 2009 it proved to be a hotbed of innovation and disruption, with a number of new storage technologies approaching mainstream adoption during the year.

One example is solid state storage. Solid state disks (SSDs), based on so-called Flash memory, are already ubiquitous in modern life: consumer electronics devices such as Apple’s iPod range exploit SSD’s lightening-fast output speeds to allow MP3s to be retrieved instantaneously. This high speed derives from the absence of any moving mechanical parts in SSDs.

However, until recently the cost of Flash memory has prevented widespread enterprise adoption. When they first became widely available in 2008, enterprise-class SSD systems cost around 40 times more than the fastest fibre channel-based systems on the market.

In the intervening period, however, the costs have plummeted. Between 2008 and 2009, the average cost of Flash-based storage systems fell by 76%.

Even as the price continues to drop, however, it is not generally believed that SSDs will be used in all circumstances. Other systems, such as SATA drives, hold functional advantages when it comes to storing archive data that is high in volume but infrequently retrieved or altered, for example. Concerns have also been raised about the reliability and durability of Flash drives, although these are refuted by some suppliers.

SSDs are better suited to applications that demand a high volume of transactions at high speed. The massively multiplayer online reality game Eve Online for example, can play host to as many as 50,000 concurrent users. Any single activity by a player constitutes a transaction with a single, centralised database that weighs in at around 1.5TB.  

As the volume of transactions began to cause system latency, CCP Games, the company behind Eve Online, replaced its IBM RAID array with an SSD device from Texas Memory Systems. “Players immediately noticed a huge performance boost across the entire game,” James Wyld, virtual worlds project leader at CCP told Information Age recently. “Shortly afterwards, we smashed our record of concurrent users.”

As well as storage uses, Flash memory is allowing enterprise system designs that were previously unimaginable. Software giant Oracle’s recently unveiled database appliance Exadata 2 contains 5TB of Flash memory. By temporarily transferring sections of a database into this Flash memory, the appliance can deliver far greater transaction speeds than conventional hardware, the company says.

De-duplication drama

The storage sector was also home to 2009’s hottest bidding war in the IT industry, which broke out in May between storage giants NetApp and EMC over storage de-duplication provider Data Domain.

Even as the storage industry as a whole saw a fall in revenues of 11% to $3.8 billion in the difficult first quarter of 2009 (according to Gartner), Data Domain – whose technology finds and removes duplicate data points in storage backups, therefore reducing the amount of storage capacity required – saw a 50% revenue increase to $79 million. Clearly, the concept of using existing storage capacity more efficiently, rather than acquiring more, struck a chord with customers.

This helps to explain why NetApp and EMC went to such great lengths to acquire a provider that effectively reduces demand for their own products, and why each company issued a number of competitive bids for it.

Eventually, EMC’s $2.1 billion cash offer won shareholder approval, and Data Domain now forms part of that company’s varied ‘information infrastructure’ product suite.

Speaking to Information Age in October 2009, Data Domain’s managing director in the UK, Phil Turner, said 2010 will see the company applying its technology to different kinds of data storage: “Our focus now is growing the backup market, and we are moving to archiving,” he said. “De-dupe as a technology has many more uses and it will ultimately march towards higher tiers of storage.”

Life cycle management

This year’s Effective IT Survey found that businesses are becoming increasingly sophisticated in their approach to managing stored data over time. The strategy of introducing ‘information life cycle management’ – using systems and processes that govern the archiving and retention of data from creation to deletion – was one that as many plan to adopt in the coming year as have done so already.

The likely reasons for this finding are twofold. Firstly, regulation governing the retention (or otherwise) of data is increasingly burdensome. In some industries, companies are required to retain data for as long as 60 years, while in others certain kinds of data must be purged as soon as it is no longer in use. This has created demand for systems that allow companies to manage their storage archiving policies more effectively, and in a more transparent and auditable fashion.

The second reason is the increasing diversity of storage media, and the growing complexity of storage systems. Besides

de-duplication, storage systems are increasingly becoming virtualised (34.0% of respondents to the Effective IT Survey have adopted storage virtualisation and 17.0% plan to in 2010), while a number of ‘cloud-based’ enterprise storage systems have been launched in the past 12 months, suggesting that the typical enterprise storage infrastructure is only going to become even more complex.

For businesses to maintain storage policies in the face of this complexity, they require storage management systems and processes divorced from the storage media itself, hence the uptake of ILM. The complexity of storage systems is also beginning to tax the capabilities of the networking infrastructure used to link such systems together. Some analysts are predicting, therefore, that 2010 will see mainstream adoption of the Fibre Channel over Ethernet protocol, which reduces the number of network switches required to handle storage workloads.

The adoption of fast, more flexible and more efficient storage technologies will, according to market analyst company IDC, have a significant impact on enterprise system design as a whole.

“The storage industry is at the beginning of a considerable transformation in which storage will move from a supportive and peripheral role to being one of an essential and fundamental pillar on which IT infrastructures will be architected,” the company said in a recent report.

That does not mean the industry will be particularly well rewarded in 2010 – IDC predicts that revenues for storage vendors will remain flat during the year.  But the uncharacteristically fertile period of innovation in the storage sector looks set to continue in earnest.