Data centers under water: What, me worry?

Overall, Gartner's Kumar insists, too few IT leaders are taking the signs of weather and climate change seriously enough. "We have had cases of coastal [flooding] where climate change has become an issue," he says. "In London and Germany, the winters seem to be getting slightly worse; we've had cases of component failure -- small bits of electrical equipment freezing up."

Despite all this, most IT managers still aren't willing to make proactive risk assessments to avert disasters. "When I mention risk assessments, they say, 'Rakesh, great point. We'll get engineering to complete a report.' But two months later, it's still at the bottom of the 'to-do' pile."

How to keep going in the next freak storm

You can do many things to plan ahead of a storm. Here are the top picks from those who learned the hard way during Superstorm Sandy.

Secure fuel tanks and pumps. In Manhattan, most downtown data center outages occurred due to ruptured fuel pumps, tanks and shorted electrical wiring. At Datagram -- a private co-location company providing managed services to Gawker, Buzzfeed and The Huffington Post, among others -- torrents of water in the basement blew off stainless steel doors, rupturing a diesel fuel tank. "Our generators were operational, but we had pumps submerged under saltwater, which caused a failure of our fuel systems," says Alex Reppen, Datagram's CEO. The lesson: Install multiple fuel tanks and encase them in concrete so they can't be damaged in a storm surge.

Check emergency wiring, hoses, connectors and back-up circuits. At Peer1 Hosting, with 21 data centers around the country, human bucket brigades brought needed diesel fuel up to a rooftop generator as multiple shifts of people worked overnight to keep a Manhattan data center online. However, "we had a heat issue and lost a portion of our data center for a day because we couldn't [reduce] the heat during the power outage," explains Ted Smith, senior vice president of Peer1 Hosting. Lessons: Review backup power plans, check UPS batteries and make sure there are enough emergency wiring, hoses, connectors and fuel to run backup equipment and generators for prolonged periods.

Get creative in the basement. When basement floors flood, fuel tanks can float, but they also do damage to pipes and fuel pumps. That was a big problem for Internap during Sandy, and New York City regulations require that fuel tanks remain in city basements. "The challenge," says Internap's Steve Orchard, "is that fuel tanks may be underground, somewhat buried, or sitting above ground in basements on stands depending on when the building was built." Lessons: Make tanks submersible by strapping them into place to structural steel so they won't float, Orchard says. "We're also looking into submersible pumps sitting inside the fuel tank along with placing a redundant pump on the mezzanine level." For added safety, Internap could also install a basement pump at a height above six feet.

Consider triple power backup. New York's CitiServ data center in Brooklyn -- a two-building project that is part of Mayor Michael Bloomberg's efforts to consolidate 55 city data centers into one -- stayed up throughout Sandy while managing the emergency 911 and 311 services. "We placed the facility in Brooklyn outside the hurricane flood zone, and we had a triple power backup strategy," which includes utility power, emergency generators and battery, says Rahul N. Merchant, New York's chief information and innovation officer. At the height of the storm, lights flickered briefly, but generator power kicked in seamlessly. A tank holding 200,000 gallons of diesel located on-site at the Brooklyn facility -- and topped off regularly -- powered everything for 45.5 hours before regular utility power was restored.

Re-patch and restore. In addition, the city's own fiber optic network, CitiNet, was used to restore 311 service when Verizon's cable vaults flooded and failed at downtown Manhattan nodes. "We were able to re-patch the 311 network by using our own fiber backbone and connecting it [directly] into the Verizon Central Office in Brooklyn," Merchant says.

Use sandbags. The bullet just missed when Sandy hammered at QTS's data center in Jersey City, N.J. "We were impacted by rising waters and high winds, and the water pushed up right to the edges of our building," says Brian Johnston, QTS's chief technology officer. However, the building was storm-proofed; the actual data center had been located on the 16th through 20th floors, with all power generation and logic on the same level except for electrical vaults and lines coming from the streets. "In the event we lose the vaults, everything we need transfers to UPS; and our fuel oil in 1,000-gallon fuel tanks along with electric pumps in the garage parking lot are also water-proofed," Johnston says. Though QTS temporarily lost utility power, backup generators took over, and no basement flooding occurred. Lesson: "Our skilled staff redirected waters with sand bags. We had wraps around any openings in the building and our parking center, so water couldn't come into parking garage or flood our vaults." Future plans: more flood blockers -- borrowed from hurricane protocols in Miami -- and training for IT staff in emergency procedures.

This story, "Data centers under water: What, me worry?" was originally published by Computerworld.

| 1 2 Page 6
ITWorld DealPost: The best in tech deals and discounts.
Shop Tech Products at Amazon