The data storage market likes to deal in big numbers. Never more so than in October 2000, when the University of California at Berkeley declared that as much information was being stored magnetically every five years or so as was accumulated throughout the 300,000 years of pre-digital human history.
For the record, the researchers estimated that the annual amount of data being stored back in 2000 was 1.7 million terabytes (1.7 exabytes) and that the data mountain was growing by half as much again each year. That means the current data horde stands at 5.7 million terabytes.
To put those figures into perspective, the academics went on to explain that a single terabyte of data is the equivalent of a stack of documents more than 16 times higher than the Empire State Building.
The falling cost of storage capacity (as measured in cost per terabyte stored) is one of the primary drivers of this growth. As user demand for holding data has soared – that raw capacity is currently running at 48% a year, according to research house IDC – suppliers have raised the pace
|
||
of innovation, helping to drive down prices and often making it easier for IT managers to simply increase their storage capacity rather than rationalise the volumes of data being stored or better utilise their existing resources. Indeed, the user perception that there is a bottomless bucket into which they can throw their data clearly encourages them to store more, and in turn leads to the need for more storage capacity.
But there is more behind the rise in enterprise storage capacity than simply the tendency to horde. The explosive growth of certain applications is having a fundamental impact.
Analyst estimates suggest that more than 600 billion email messages are sent each year worldwide. For security or legislative or other reasons, a large proportion of those are stored, often needlessly. But if the challenges of email archiving are vast, those associated with other key IT applications, including data warehouses, records management, ecommerce, customer relationship management (CRM) and enterprise resource planning (ERP), are scarcely smaller.
CRM applications, and in particular those used in retail or in contact centres, can generate staggering volumes of data. The world’s largest retail chain Wal-Mart, for example, captures and stores every single transaction that passes through its tills, amassing over 300 terabytes of data for analysis – a data warehouse that is refreshed every 10 minutes and that has grown by a factor of 600 over the past 14 years.
On a smaller scale, but still astonishing, UK home shopping company JD Williams has grown its data warehouse from 10 gigabytes to 2 terabytes over the past dozen years. With a different application requirement, call centre operators, such as travel firm Thomas Cook, digitally store recordings of all their customer calls, retaining those that result in a sale for 18 months in case customer disputes arise.
These are just the tip of the iceberg. A recent survey by the European arm of the Storage Networking Industry Association (SNIA) found that storage capacity to handle databases was rising at 73%, email storage was growing at 63%, CRM storage at 28% and ecommerce data at 22%.
The SNIA study also found that, during 2002, almost one-third of businesses had doubled their storage requirements, while 9% had experienced a tripling of growth. “The continuing explosion of information overload from databases and email shows no sign of abating in the near future. IT directors are facing a constant battle to contain and effectively manage this information,” says Paul Talbut, chairman of the SNIA Europe.
Winning the war
A constant battle it may be, but it is a winnable war. The traditional solution of simply throwing more disk capacity at the problem is giving way to the deployment of storage area networks that ensure better utilisation of existing resources – especially at large organisations.
Companies are also training employees and enforcing policies to discourage inefficient use of storage. Take the habit of attaching big files to email messages. This is often unnecessary, since new messaging applications now enable documents to be sent as ‘feather-light’ hyperlinks, says Mark Reynolds,
|
||||
the UK managing director of TopCall, a unified messaging systems supplier. “Companies must urgently address the attachment-addicted work culture and lighten the load of communication,” he says.
Capping file sizes and compelling employees to permanently delete emails would also help, although such solutions are often seen as ‘fire-fighting’, a waste of employee time and may even conflict with records-retention rules.
Some are worse offenders than others. A customer survey by Waterford Technologies, an email-management software developer, found that about 1% of the workforce typically is responsible for using up about 20% of the available email storage space. At one client, about 90% of all email was of a personal nature – a situation exacerbated by the rapid rise in employees using company storage resources to hold downloaded music files and their digital pictures.
Not everyone has answered the call to action, though. As many as one-third of UK companies do not have a formal email usage policy, according to research commissioned by back up software company Legato.
Forward-thinkers are implementing email usage and storage policies, however. Until recently, the roughly 220 employees at Britannic Retirement Solutions, a financial services subsidiary of the UK’s Britannic Group, had to make do with a maximum of ‘only’ 45 gigabytes of email. Users struggled to keep within those limits and the company found it took hours to back up messages.
Peter Goadby, the company’s IT director, says he chose to invest in an email archiving system from Sagitta Performance Systems, a data-management company, not only to ease this pain but also to ensure that the company met its regulatory requirements. “We recognised the need for reliable back-up of all business communications, but due to sheer storage volumes we were struggling to manage within the allocated backup window,” he says. Sagitta estimates that its software can reduce email back-up times by up to 25%.
Automating the process of managing the movement of data around the storage infrastructure is another key step. Paul Hammond, solutions consulting director of CNT, a storage networking specialist, says that organisations need to establish a ‘rules-based’ approach, whereby different types of data are automatically assigned to different types of storage device, depending on its value to the business.
“The storage landscape is changing,” he says. “Taking a more policy-driven approach – applying rules that dictate the treatment of data as and when it is generated – should go some way to helping businesses overcome the ever-increasing data mountain.” Hopefully before they find themselves overshadowed by ‘data skyscrapers’.
|
|||||
