Four tips to prepare for the next Amazon outage
It happened again: Amazon
Web Service suffered its latest outage in late June. Now, as the dust has settled, customers are reassessing what
lessons can be learned and how to prepare for the inevitable next one.
Compared to AWS's major outage last
summer, which was caused by human error and resulted in an overloaded network, the most recent incident
resulted from an electrical storm causing a power outage in an AWS Virginia data center. While the actual outage
only lasted about 20 minutes, the domino effect of a backup generator not kicking in, combined with software bugs
AWS had not seen before, caused about 7% of customers in the impacted area to be down, some for as much as three
hours on the evening of Friday, June 29.
COMPETITION HEATING UP: Amazon in the crosshairs of
Google and Microsoft
As storms ripped through the mid-Atlantic coast that Friday night and into Saturday morning, parts of sites such
as Netflix, Pinterest and Instagram were down, sometimes for as much as three hours. But it didn't have to be that
way. Software startup Newvem tracks AWS customer usage, and officials say misconfigurations by customers
exacerbated the problem on that Friday night. Newvem and Netflix have four suggestions of how the latest outage
could have been mitigated and how to prepare for future incidents.
1: Use snapshots
Backing up data is critically important to ensure high availability and AWS gives customers the option of
backing up their Elastic Block Store (EBS), which is a file storage
service impacted during the latest outage, with a "snapshot." EBS Snapshots make a copy of the EBS volume and back
it up in Amazon's accompanying Simple Storage Service (S3) offering. User to have to initially back up their entire
EBS volume to S3, but then whenever there is a change to the content of the EBS volume, only the new data has to be
captured in another snapshot for the volume to be recreated. Of Newvem's more than 500 customers, 45% of users who
have large AWS clouds, meaning those with more than 101 instances, did not have effective EBS snapshots.
2. Ensure correct ELB configurations
One of the advantages of using Elastic Load Balancers (ELBs) is they can automatically reroute traffic based on
availability and need. But Newvem found that up to 20% of heavy users aren't properly configuring their ELBs
either. One of the most common misconfigurations is to reroute ELB traffic within the same availability zone (AZ).
AWS has multiple availability zones within its regions, which are meant to be isolated from one another. By not
configuring the ELB to route traffic to a separate AZ, users aren't protected if their AZ is impacted, Newvem
3. Test, test, test
One of the bigger names that went down during the latest AWS outage was Netflix, which during the past few years
has migrated much of the company's video streaming services to the AWS cloud. During the latest outage, the site
had selective service disruptions between 8 and 11 p.m. PDT on the Friday night of the outage.
In response, Netflix wrote a blog post outlining
changes they will make to prepare for an AWS disruption and area they hope to ramp up is testing. Netflix already
has "Chaos Monkey," which simulates an outage of random instances within the Netflix AWS cloud. But that's
apparently not good enough. The company is now developing a "Chaos Gorilla," which will simulate an entire
availability zone going down to ensure the system can automatically handle the situation.
4. Not just multi-AZ, but multi-region
After last year's outage, AWS officials reminded users that
using multiple availability zones is the best way to ensure AWS cloud resiliency. Now, Netflix and Newvem officials
agree that instead of a multiple AZ architecture, spreading workloads across multiple regions, or even across
multiple cloud providers is the best way to ensure high availability. "Using multiple regions is really the new
best practices for customers that really require high availability," Newvem CEO Zev Laderman says. Netflix says as
it is expanding its global footprint to allow streaming of its video content around the world, it will be moving to
a multi-region support system as well.
Network World staff writer Brandon Butler covers cloud computing and social collaboration. He can be reached at
BButler@nww.com and found on Twitter at @BButlerNWW.