The signs of an impending data crisis might once have been apparent from the abundance of overflowing filing cabinets. Today, data grows almost invisibly inside the enterprise, a virtual avalanche threatening to overwhelm information flow and undermining decision-making.
The issue of how businesses effortlessly amass data is reasonably well understood within IT organisations. Email – along with corporate instant messaging and increasingly, IP telephony – has added to the already bloated volume of documents being collected: from databases and word processing documents, to spreadsheets and PowerPoint presentations – each different form adding to the burden. No matter how clearly the phenomenon of data growth is perceived, the issue of how best to tackle its consequences remains opaque.
The extent of the problem is highlighted by the views of a senior storage architect from one of the UK’s largest engineering companies. Speaking at a recent Information Age off-the-record debate, he outlined how the “smart” engineers at his company had developed a highly sophisticated equipment-monitoring machine, capable of analysing the performance of vital assets in detail. “The problem is, each time it’s used, it generates over 1 terabyte of data, which [the IT department] have to store. And they’ve no idea what to do with all that data,” he said.
The knock-on effect of this kind of demand is being widely felt. At one business service organisation, spending on disaster recovery – as a result of data replication and offsite storage – has risen in line with exponential growth in data volumes, said its director of technology. That has cut directly into the budget at the expense of more innovative and value-adding IT projects.
Many delegates also said that they were already running out of data centre space; facilities expected to last for years were already nearing capacity.
Reply to all
Regulatory compliance issues are exacerbating the situation even further, as the head of systems management at a multinational bank explained: “Regulation has meant that everybody feels compelled to keep everything forever.”
And even the most seemingly benign of applications can add horrendously to the volumes of data being created. A single email is of almost inconsequential size on its own. Once an attachment is added, the size rises several-fold; if this is then sent to multiple users, where one of whom may then choose a hit the ‘reply to all’ option, the volume of data being stored on desktops and at the email server begins to increase exponentially, while adding little value to the business. One delegate recounted how a large bank had calculated the impact of this with just a single email: it reckoned it had incurred £200,000 in storage after distributing an updated fire policy to employees via an email.
But while the consensus around the table reached as far as agreeing on the scale of the problem, strategies for alleviating the issue created much dissent. For some, the nightmare of unconstrained email archives is best dealt with harshly: impose strict limits on inbox sizes and force the management and maintenance on to the end user.
Others were aghast at this prospect. “Our business depends on storage-hungry users. If they want another terabyte, I’ve got to give it to them,” said one participant.
Delegates all agreed, however, that the current rampant growth in data is not sustainable, the need for action paramount. “If data is growing 100% year on year, in 10 years’ time, your business will be paralysed. No business can afford this.”
