Severe weather intensifies focus on disaster planning
Companies should no longer depend on only a cloud provider for backup
Severe thunderstorms knocked out power to 1.2 million homes in the D.C. area. Wildfires ravaged more than 2
million acres in the Rockies. Two-thirds of the country is in drought conditions, and flooding is expected to get
worse as the time between rainstorms lengthen and, in turn, grow more intense.
Intensifying weather patterns threaten businesses as global warming raises the temperatures of the oceans.
Disaster recovery plans that include only backing up data regionally may need to be rethought, experts say.
The cloud, which was supposed to guarantee high availability, has been significantly affected by power outages
caused by storms. Companies dependent on cloud service provider Amazon Web Services (AWS) found their websites down
a week ago when rare severe thunderstorms, known as derechos, struck the Virginia and Washington D.C.
area, leaving 1.2 million homes without power for days.
"We thought that by simply deploying [our website] across multiple active zones on Amazon we were going to have
the backup we needed," said Brandon Wade, CEO of WhatsYourPrice.com, an online dating site with 400,000 active
members. Wade said his site went down twice, for two hours each time after Amazon lost power. More than 1,000
customers contacted WhatsYourPrice.com to complain, he said.
WhatsYourPrice.com had initially signed on with Amazon's AWS 18 months ago because using a service provider
reduced costs for the start-up. But, after two outages at Amazon in less than a month, WhatsYourPrice.com
canceled the cloud service, and is now deploying its servers in two co-location facilities, as well as using a
local Las Vegas cloud provider for data
"If your business can take a few hours of outage when a disaster strikes, then [Amazon AWS] is the solution to
have," Wade said. "Otherwise, you need to architect in a smarter way."
Derechos, which are caused by severe heat waves and can develop hurricane-force winds, can span hundreds of
miles. The storm on June 29 spanned some 700 miles and had average wind speeds of 60 mph.
An Amazon spokesperson said the derecho caused AWS to lose
primary and backup generator power to a portion of a single "Availability Zone" in its US-East Region on June
"In the thunderstorm on Friday night, several of our data centers had their utility power impacted, but in only
one of them did the redundant power not operate correctly (which ended up impacting a single digit percentage of
our Amazon EC2 instances in the US-East Region). We began restoring service to most of the impacted customers
Friday night, and the remainder were restored on Saturday," the spokesperson wrote in an email response to
A firefighter works a burnout operation on the north flank of the Fontenelle Fire outside Big Piney, Wyoming.
More intensive natural disasters such as wildfires and derechos are requiring companies to pay more attention to
disaster recovery planning. (Photo: Jim Urquhart / Reuters)
Netflix, Instagram, and Pinterest websites were also affected by Amazon's recent outage.
"Netflix could have had Amazon duplicate its instance in every data center, but that would be prohibitively expensive,"
said Dan Olds, an analyst at Gabriel Consulting Group. "But, expect to pay more for that kind of resiliency.
There's a price to be paid for that kind of availability and there's no free lunch in the cloud."
While cloud data services are still in their infancy, eventually they will become so common that they'll be
considered a utility, just as electricity and telephony is today, said Parveen Jain, CEO of RedSeal Networks, which
sells software for monitoring the health of companies' IT infrastructure.
"I think the service providers have to make sure systems work uninterrupted as much as possible. They have to
implement redundancy," Jain said. "As the government is transitioning to the cloud, and some of the largest
enterprises are moving there as well, redundancy is a necessity."
Power outages like the one Amazon's AWS experienced are expected to become more frequent as global warming
causes storms of all types to become more severe and heat waves to be more frequent and intense.
According to the U.S. Drought Monitor, administered by the National
Oceanic and Atmospheric Administration (NOAA), the vast majority of the U.S. is experiencing drought
So far this year, more than 23,000 high temperature records have been set, compared with 2,500 cold temperature
records, according to NOAA. In past years, records for
high and low temperatures were typically equal, said Harold Brooks, a research meteorologist at the National Severe
Storms Laboratory in Norman, Okla., which is also part of NOAA.
Brooks said global warming has been responsible for only about a half degree of temperature increase on average,
but it doesn't take much of a temperature change to have an effect on weather conditions. And temperatures are
expected to continue to rise over the next century.
"We're very confident that global temperatures have increased. That's a slam dunk," Brooks said. "And we can
tell there are regional temperatures that have gone up significantly. We know we're setting a lot more high
temperature records than low temperature records.
"When it rains, the rains will be more intense, but there will be longer gaps between rains," Brooks said. "The
overall amount of rain has slightly gone up in much of the U.S. It's getting more concentrated into heavier rain
There is also meteorological modeling and "empirical evidence" that shows if nothing else changes, the strongest
hurricanes will get stronger, according to Brooks.
"There are indications that wind speed and damages are tied pretty closely together. If we're talking a few
meters per second more in wind speed, that could lead to 10% to 20% more damage. We may not notice the wind change
all that much, but we may notice more damage," he said.
The 'hugability' factor
"Last year was the worst year we've had in the history of disasters," said Al Berman, executive director of the
Disaster Recovery Institute (DRI). "I've been doing this since the 1990s. I started out hearing this was going to
be a very calm summer, and now I'm hearing it's not going to be." The DRI is a nonprofit organization that provides
educational services and certification for contingency planning and business continuity professionals.
Berman said one thing he still can't understand is why companies insist on maintaining an IT presence in a
single building - something he calls the "hugability" factor. Even the largest banks, he said, like to have their
technology in the same building as their business operations. That kind of proximity, however, means a single
regional disaster can take out a company's entire infrastructure.
"If they were just smart enough to move their servers from one building to another, they'd avoid these regional
issues," he said.
At far greater risk, however, are small- and medium-sized businesses (SMBs). The SMB market, which is made up of
33 million companies, represents the sector where most U.S. workers are employed, yet it's the sector that's least
prepared to respond to disasters because it has far fewer resources.
Cloud backup providers, such as Carbonite, Box.net and Dropbox, are good alternatives to building out a
company's backup infrastructure, Berman said.
Aging infrastructure also adds risk as more severe weather strikes various regions of the U.S. In New York City,
for example, many water pipes date back to the Civil War era, according to Berman.
In addition, the nation's power infrastructure is mainly above ground, on poles that are vulnerable to strong
winds and electrical storms.
"There is some infrastructure that needs to be changed. I think burying power and phone lines is a good first
step," Olds said.
Utility companies should be spending time and money to bury lines after severe storms, in a piecemeal fashion,
Olds said. That way, they'll have both the political and financial support for the projects. Unfortunately, after a
power outage, most local governments are only concerned with getting power back on and not addressing future
outages, he said.
The nation's communications networks are in even more critical need of changes, Olds said. For example, as a
result of the mid-Atlantic coast derecho, 911 emergency service was out in areas of Virginia for as long as two
days, Olds said.
Installing fiber-optic communication lines underground could ensure not only more resilient communications
networks, but better coordination during and after severe weather events. And, the use of new mini-sensors on
communications and power networks could lead to faster line repairs, he said.
"By installing network sensors, you can see where the damage is and deliver resources to the right areas,
faster," Olds said. "It's getting cheap to put sensors and intelligence into anything."
Until the nation's infrastructure is made more robust, the increasing severity of weather conditions will
continue to have regional, not just local, implications. All companies need to think about backup plans outside of
"I advise my clients that you can't depend on any one thing. You can't depend on the cloud unless you have
something to fall back on," Olds said. "I also hope this serves as a great learning experience for Amazon and other
cloud service providers in general."
Lucas Mearian covers storage, disaster recovery and business continuity, financial services infrastructure and
health care IT for Computerworld. Follow Lucas on Twitter at @lucasmearian or subscribe to Lucas's RSS feed. His e-mail address is firstname.lastname@example.org.
Read more about business continuity
in Computerworld's Business Continuity Topic Center.