Reading the news yesterday, this article on CNN.com caught my eye. I mean, the headline – Solar Explosions Could Impact Earth – is assuredly alarming. After reading it, however, I concluded that I could stand to lose the GPS capability on my smart phone and be none the worse off.
But then I saw reports about how these explosions – solar flares which are giant explosions on the sun that send energy, light and high speed particles into space, according to NASA – could wreak havoc on data centers. And sure enough, on NASA’s site, particularly high-powered flares directed at Earth, such flares and associated solar magnetic storms known as coronal mass ejections (CMEs) can “create long lasting radiation storms that can harm satellites, communications systems, and even ground-based technologies and power grids,” according to this article.
NASA and others (such as the National Oceanic and Atmospheric Administration) report that they are vigilantly monitoring for X-class flares and their associated magnetic storms, so there is that. But still, the solar flares remind us that Mother Nature is a force to be reckoned with.
So is, apparently, lightening. Just ask Amazon, which has yet to fully restore customer data lost because of an outage at a data center in Ireland caused by lightening. You can read about this here on our sister publication, Network World. Amazon is reportedly delivering recovery snapshots to affected customer accounts, but the lightning strike and subsequent outage illuminate just how vital disaster recovery plans are.
Apparently, the lightning strike took out the both the primary and secondary power supplies of Amazon’s Dublin-based data center. The strike caused a transformer explosion and fire in the grid of Amazon's electricity supplier and knocked out Amazon's backup generators. On its EC2 cloud's Service Health Dashboard for users, Amazon reported that the strike caused an “electric deviation” that traveled along the power feed wires and then knocked out the control system that would normally have triggered backup generators in the data center, thus requiring Amazon operators to manually synch the process to start the generators.
The lightning strike on Amazon’s data center apparently was a triple whammy. Not only did it hit the electricity supplier, it hit primary and secondary power supplies and delayed the backup generators. It may be that there was little in the way of preventing the outage from a power perspective. The Uptime Institute does provide some advice in this article here; it’s worth a read.
There’s more to a solid disaster recovery plan than power supplies of course. For example, once the power goes down, automated fail-overs to different sites can help stem data loss. And even then, there may be a bit of data loss unless there’s real-time data replication between the various sites.
I’d love to hear from readers about their best practices for protecting data centers against lightning. And, of course, solar explosions. Got any (best practices, not explosions)?