November 02, 2012, 9:32 PM — The classic quote from Warren Buffett about a financial crisis is also true for data centers. It's "only when the tide goes out do you discover who's been swimming naked," the multibillionaire businessman said.
Hurricane Sandy brought the tide in, literally. But the flooding in lower Manhattan had the same effect on data centers that, as Buffet observed, a recession has on financial services. It exposed the problems and shortcomings in the system.
There were data center outages. The causes range from flooded buildings that cut data centers off from fuel supplies, to backup generators that malfunctioned. The problems were extensive and the responses were often dramatic.
Zayo, which runs a data center in one of the most wired buildings in New York City, at 111 8th Ave., sent an "urgent communication" to customers Thursday. The temperature in one of its data center suites had reached 93 degrees Fahrenheit. Another communication soon followed, reporting that temps had "risen above 100 degrees."
Zayo conducted continuous conference calls for customers, detailed its multiple paths to a fix, including estimating how long it would take to get a 2MW backup generator delivered through New York traffic. The problem was a result of a failed isolation valve in the fuel system supplying a generator. It was fixed without any customer outages, and temperatures returned to 70 degrees.
In one case, at 75 Broad St., the response was just amazing.
Peer1, a data center that operates at 75 Broad St., was close to shutting down its data center after water flooded a basement, cutting off access to generator fuel supplies. That's when the employees of two customers of Peer1, Squarespace and Fog Creek Software, stepped in to carry fuel via a bucket brigade, up 17 flights.
There were many other reports of data center problems, particularly after the initial storm impact. Early in the week, Internap reported a data center outage at 111 8th Ave. due to a building-fed fuel system problem. Equinix reported a failed generator that "impacted service to several customers." The problems were eventually fixed at both sites.
New York's problems affected customers far and wide. Web sites were inaccessible and some networks were down.