Storms, leap second trigger weekend of outages

A number of high profile websites suffered outages this weekend, as cloud computing provider Amazon Web Services was beset by power outages and the "leap second" triggered various software errors.

Late on Friday night, Amazon Web Services revealed that "a large number of instances" on its EC2 cloud computing service, hosted at its North Virginia data centre lost power as a result of electrical storms. The outage also affected the AWS load balancers, which route traffic to server based on demand and availability.

Power was restored to the facility within half and hour of the issue first being confirmed, but the company was still working to restore some instances 12 hours later.     

Affected customers included online film service Netflix, social website Pinterest and Instagram, the mobile phone photo app acquired by Facebook for $1 billion.

On Saturday, websites including social link sharing site Reddit reported technical issues following the "leap second" – an extra second on June 30th inserted to synch ‘atomic time’ with the rotation of the earth.

The bug seemed to affect the Java application platform and the Linux operating system. Reddit said it was suffering "Java/Cassandra issues" (referring to the non-relational database system), while browser make Mozilla said it suffered issue with big data platform Hadoop, also built on Java.

Last year, web giant Google revealed how it deals with leap seconds, using what it calls a "leap smear".

"We modified our internal [Network Time Protocol] servers to gradually add a couple of milliseconds to every update, varying over a time window before the moment when the leap second actually happens," site reliability engineer Christopher Pascoe wrote in a company blog post. "This meant that when it became time to add an extra second at midnight, our clocks had already taken this into account, by skewing the time over the course of the day."

Earlier this year, Microsoft’s cloud service Azure suffered an outage lasting up to 23 hours following the "leap day", February 29th.

Avatar photo

Ben Rossi

Ben was Vitesse Media's editorial director, leading content creation and editorial strategy across all Vitesse products, including its market-leading B2B and consumer magazines, websites, research and...

Related Topics