Huge customer effort keeps flooded NYC data center running

How do you get hundreds of gallons of fuel up 17 stories? Your customers carry it

By , Computerworld |  Data Center, Hurricane Sandy

The generator-fueling effort got organized. More people begin showing up from Squarespace and another large user, Fog Creek Software. They included software developers, systems engineers, sales representatives, executives and support personnel.

A network of people, stationed at every staircase landing, formed a bucket brigade passing fuel from one person to the next. Several dozen may have been involved over multiple shifts, including some day laborers who were also hired to help.

"Over the next night into the morning, we were able to successfully continue to coordinate fuel trucks coming and manually move hundreds of gallons of diesel to the roof and keep the thing online," Casalena said.

There were moments when they felt they weren't moving fast enough. At one point, a generator light came on showing low fuel.

"It's kind of just very dramatic," said Casalena. But they managed thanks to a "huge group effort," he said.

"It looks like we have operationalized this to the point where we can make it work -- I can't honestly believe it," said Casalena, "It somehow worked."

They now have enough fuel on the roof to stagger deliveries.

"Morale was super high," said Joel Spolsky, the CEO and co-founder of Fog Creek Software, who runs a number of businesses out of the Peer1 data center. "People were just absolutely itching to get there and help and do something and make some kind of contribution."

"It was hard, but people were kind of excited to have a mission to work on," said Spolsky, who is also co-founder of StackExchange, a major site for developers. That operation was moved to a data center in Oregon, but Spolsky has other work in the Peer1 center that can't be easily moved.

Spolsky, who tweeted about the efforts, said he even had offers from customers and people on Twitter to help out. "I think people feel kind of helpless sitting at home watching the news, and they were so excited to be able to do something even if it was hard to make a substantial contribution," he said.

The efforts continued to restore the basement pumping system. Water kept finding its way into the basement, but Casalena said that as of early Wednesday evening, it appeared the basement flooding situation was getting under control.

Both Casalena and Spolsky praised the Peer1 New York operations staff for their efforts. Peer1, which has 19 data centers worldwide, has extra help arriving tonight, driving up from a Virginia because of problems getting flights to New York.


Originally published on Computerworld |  Click here to read the original story.
Join us:
Facebook

Twitter

Pinterest

Tumblr

LinkedIn

Google+

Answers - Powered by ITworld

ITworld Answers helps you solve problems and share expertise. Ask a question or take a crack at answering the new questions below.

Join us:
Facebook

Twitter

Pinterest

Tumblr

LinkedIn

Google+

Ask a Question