The Avengers hit the home video market Tuesday. It was the most pre-ordered movie in Amazon’s history, and if you ordered directly and digitally from Amazon, it was mere milliseconds between transaction and being able to watch the Hulk sucker-punch Thor. Amazon has The Avengers on row after row of servers, in building after building in Northern Virginia, ready to stream to you as fast as you can receive its Marvel-y excitement. The price? $14.99, but the New York Times would like you to consider the batteries, diesel, disk space, and national power grid modifications necessary for that instant video.
The Times made the case for considering, or even shaming, the physical impact of “the cloud” in its front-center story this past Sunday. It’s roughly 4,400 words, based on what writer James Glanz describes as a year-long study of data centers and a commissioned survey of server utilization. And the article does pretend it doesn’t have a message, and that message is: all the things we’ve come to expect from the web and from apps are hugely wasteful, to the tune of “6 percent to 12 percent” utilization, much the same as in 2008. And we’re all demanding more and more from these presumably inefficient servers.
The reaction? As swift and efficient as Iron Man’s HD aerial maneuvers. Diego Doval, computer scientist and former CTO for Ning, describes the article as “a mix of half-guesses, contradictions, and flat-out incorrect information.” Katie Fehrenbacher, who wrote a four-part series on data centers for GigaOm, writes that it seems as if Glanz “jumped into a time machine and did his reporting a couple years ago.” Wired wants to remind us that the data center teams at Google, Amazon, Facebook, and Apple are very secretive and keeping their advancements to themselves—”The stuff that companies will talk about is often so out-of-date that it’s obsolete.” Almost every critic who took the time pointed to Glanz’s poorly chosen, six-year-old anecdote about engineers buying fans at Walgreens to keep Facebook from melting, and they were right to do so.
To summarize the backlash: the Times has no clue how servers should be measured, and they can’t measure the largest and most innovative players in the field, so why did they bother even trying to talk about our field?
There is, however, a midpoint between ”get a clue” and “Quick, call a Congressman.” And you, a smartphone-toting, Gmail-searching, photo-uploading tech enthusiast, should think about it. As David Cappuccio, chief of research at Gartner, puts it in the very last paragraph of the Times article:
“That’s what’s driving that massive growth — the end-user expectation of anything, anytime, anywhere. ... We’re what’s causing the problem.”
Millions upon millions of dollars and research hours have gone into one side of the power plug. Amazon’s data services power so, so many web and app-based startups. From just the narrow window given to its customers, Amazon appears obsessed with efficiency and on-demand scaling, at a level beyond what almost any web-facing project could manage without an entire full-time department. The grid that powers those servers, however, and the cooling required to keep them running if they are truly utilized efficiently, are not subject to tweaks a crafty engineer can make while remotely connected from home. You can submerge your servers in oil, but you can’t smooth out the power demands of keeping everything available all the time.
To keep itself prepared, and to prevent outages that occurred in 2009 and 2011, Amazon, like most data center operators, must keep not only massive backup batteries on hand, but diesel generators. Those generators produce fumes, and must be test-fired regularly to ensure readiness. Amazon has learned to be redundant and cautious, after minor incidents like a short on an electrical pole triggered domino-like chain reactions, leading to customers who pay Amazon for peace of mind having to develop fail-safe failure modes on the fly.
That’s how a remarkably efficient, solutions-minded firm like Amazon handles data reliability. Smaller firms, and companies for whom data isn’t a central business but a necessary utility, fare much, much worse in efficiency. Paying a premium to third-party data centers for their own power drain and utilization rates, at the levels that were likely the source for the Times’ cited 6-12 percent utilization rate, is something too many businesses are doing.
So what about you, the person who shoots photos, saves emails, and checks social networks—what can you do? If you read The Omnivore’s Dilemma or watch Food Inc., you can make changes in your shopping and diet to act local, think global, and play your part. But what if you’re suddenly struck by a feeling that you could be using the cloud in more ecologically friendly fashion?
You probably can’t run your own cloud storage and backup system more efficiently than Amazon. And you can only go so far in demanding to live your life with entirely localized data, because there are just too many interesting and useful things out on the web to ignore out of puritanical zeal. But you can do something.
I think it’s smart, for example, to have something like a local-but-cloud-like backup in place, such as Pogoplug (which, for some plans, optionally uses Amazon’s Glacier storage service). And you can do your best to ensure that anywhere you’re storing your data is forthcoming about where it’s being stored, and how efficiently. You might no be able to move your company’s data structures over to Amazon or another efficient firm next week, but with your personal data, you can make sure you’re not keeping data on servers for no reason other than, “Hey, it’s free.”
Because if there’s anything the Times server efficiency manifesto got right, it’s that there’s a notable jump disconnect between “in the cloud” and the reality, which is “on the drives of somebody I hope is thinking big-picture.”