IT disaster recovery: flooding lessons learned

We all know that computers and water don’t go together. So every year when we see the news that whole towns or regions in the UK are flooded – whether it’s York, Manchester, Leeds or Cumbria – CIOs, CTOs and IT managers should feel shivers going down their spines.

Still, (according to research by Zenium Technology Partners) half of all UK companies operating data centre infrastructure admit that they are unprepared for natural disasters; in other words, extreme weather events have the potential to cause massive problems for their company’s operations.

Take Vodafone’s data centre in Leeds: it was caught up in last winter’s floods in Northern England; the damage left customers with intermittent voice and data services.

The total flood damage across Northern England topped more than £5bn.

For IT managers, relocating infrastructure is one option, of course.

However, this is usually not only costly and time consuming; it also provides no guarantee of safety – unless you run two data centres in parallel to safeguard against any data loss.

>See also: Back-up blunders suggest employee confidence in data recovery could be misplaced

When Solihull Metropolitan Borough Council decided last year to relocate its data centre, the city reportedly spent £1 million in the process.

However, there are alternatives. CIOs, CTOs and IT managers can prepare for potential disaster by working closely with experienced infrastructure architects and data recovery experts.

They need to understand which vital business processes need to be protected and how they can be recovered in the event of flooding.

Good planning will put effective solutions in place that can recover all business processes within the required time frame, thanks to data records being stored off-site.

Should an emergency situation occur at any business location, the plan also has to outline a process for IT management to be proactive and control all recovery activities.

Companies can’t develop such a process on the fly, not least as they have to do more than just solve their own problem; they also have to think through what negative impact their data centre failure could have on stakeholders.

A case study of how natural disaster can strike IT, and how to recover, is last year’s floods in Southeast India.

During December, the region experienced its heaviest rain for a century, with more than 1,000mm of rainfall.

The severe weather resulted in the flooding of the runway and apron of Chennai Airport, as well as loss of power to the terminal.

This had knock on implications way beyond the airport, because Chennai’s IT infrastructure supported not just the local airport, but also a further nine airports across the country, which were brought to a standstill.

>See also: How to do disaster recovery without disrupting your business operations

However, the authority that runs these airports had prepared for such an event by working with NIIT Technologies, to develop and prepare a comprehensive disaster recovery plan.

As a result nine out of 10 airports were able to come back online within just 15 minutes, as the data centre operation in Chennai was instantly transferred to the data recovery centre in Kolkata.

Thanks to the this disaster recovery programme, network outages could quickly be isolated, even though the flooded runway meant that flying operations could not resume at Chennai Airport.

The disaster recovery plan allowed internet connection to be rerouted, creating emergency VPN connections to enable operations and avoid further complications.

To protect businesses against potential losses and to ensure a smooth recovery process in the event of a natural disaster, the disaster recovery experts at NIIT Technologies have five tips for businesses.

Critical application assessment and disaster recovery plan

A comprehensive evaluation of potential disaster scenarios – included flooding, earthquakes and power cuts – should be reviewed.

Following the assessment, a plan should be developed to document the responsibilities, procedures and checklists that will be used to manage and control the situation following an emergency or crisis occurrence.

Implementation Procedure

As part of this disaster recovery plan, critical information systems have to be backed up in real-time offsite.

>See also: The 5 imperatives for disaster recovery planning

These processes are a critical part of avoiding downtime and potential data loss.

Testing

The disaster recovery plan has to be tested regularly to ensure the business has the ability to continue any critical business processes in the event of a disaster.

This will involve regularly examining parameters, reviewing management objectives and evaluating measurement criteria and methodology, while also validating timings to examine the overall effectiveness of the current disaster recovery plan.

Maintenance of the Plan

Disaster recovery plan maintenance is of utmost importance; have the latest business processes been included in the plan, so that they can be recovered? Are all knock-on effects of a process failure well understood? Are the list of stakeholders and the procedures governing recovery up-to-date?

The recovery plans needs to be kept in sync with business changes.

Recovery procedure

During a disaster, recovery activities will be conducted in a phased approach, emphasising critical applications effectively and efficiently.

Phase I

Move operations to the disaster recovery backup site and the emergency operations centre.

Phase II

To recover critical business functions, we restore critical applications and critical network connectivity in our Disaster Recovery Sites first. Recover the system and network are crucial for your business continuity.

Phase III

Return data processing activities to the primary facilities or an alternative location.

>See also: How organisations can take a holistic approach to disaster recovery

Businesses are confronted with a broad range of disasters each and every day.

Only those that have adequately developed practised and well maintained contingency plans will survive.

Without sufficient processes in place, businesses are playing Russian roulette and risk not only losing revenue and critical data, but also severely damage their customer relations.

Yet many organisations take IT infrastructure for granted, relying on outdated disaster recovery plans or are simply complacent when it comes to maintaining them.

If a business is to achieve longevity, then strategic planning is essential.

Involve outside infrastructure architects and disaster recovery experts to independently assess and stress test your contingency plans.

Today, data centres are often the beating heart of most companies. Without a disaster recovery plan, companies may not survive the next natural disaster.

Sourced by Arvind Mehrotra, president and global business head – infrastructure management services – and Satya Samal, executive vice president and head of Europe at NIIT Technologies

Avatar photo

Nick Ismail

Nick Ismail is the editor for Information Age. He has a particular interest in smart technologies, AI and cyber security.

Related Topics

Disaster Recovery