For Martin Gill, IT director of charity telethon organisers Comic Relief, the challenge is not to provide a constant level of service throughout the year, but to ensure that one evening of extremely high traffic every two years goes off without a single hitch, and without draining significant amounts of the charity’s limited resources.
“We’ve got one moment to get it right,” says Gill. “We can’t go over budget, or cause any delays. For this reason, we have to use the best products available to us.” With feeds of transactional data coming in from the charity’s web site, from interactive TV sets and from numerous call centres, a rock solid infrastructure is essential.
“At the peak of the evening, between 9:30 and 11:30, we might get between 10,000 and 15,000 transactions every minute. Downtime of even a minute could lose Comic Relief thousands,” says Gill.
Comic Relief looked to Oracle’s 10g storage management system and DataGuard product to provide the fail-proof database availability that it required. “We have two data centres which are 8km apart,” says Gill. “On the night, one is used to log transaction data from the web site, and the other for interactive TV and call centre information. Should one be rendered unusable, we need the other one to take the extra traffic.”
To support this flexibility in an emergency and to enable the infrastructure to ‘fold both ways’, there must be a copy of all transactional data available on both sites. But unlike many organisations that might be able to synchronise data stores during periods of low business activity, Comic Relief has to maintain this synchronisation during the peak of site traffic.
Oracle’s Dataguard keeps the two data stores consistent – not by backing up the entire database at regular intervals, but by synchronising only the data that has changed. It can work over both IP networks and dedicated cabling, a flexibility that posed a dilemma for Comic Relief.
“Two years ago,” explains Jonathan Miller, director of strategic account development at Oracle UK, “Comic Relief established a stretched cluster to ensure that data was the same on both sites. The centres were connected by ‘dark fibre’, which ensured that the synchronisation process would never interfere with the traffic coming in from the Internet and call centres.”
Dark fibre (leasible fibre-optic cable networks), however, is an expensive resource. Martin Gill had to weigh up whether this link was worth the extra expense to the charity, and this year decided that it was not. “With the IP solution there is a bandwidth issue,” he explains. “There is about a 15 to 30 second lag when the synchronisation occurs. However, we decided that for some donors to experience delays when giving money on the Internet was a small price to pay to ensure that the transaction data in our two sites was safely backed up.”
Comic Relief receives software and support from companies such as Oracle, who see it as an opportunity to experiment and develop best practices while doing some good. The advantage to the charity’s two year life cycle is an extended period of debriefing during which it can assess what worked and what did not.
