Data centers under water: What, me worry?

Given the dire warnings about climate change, some business and IT people are pondering this question: How should data center managers handle the crop of 100- and even 500-year storms, coastal flooding and other ecological disasters that climatologists predict are heading our way?

Some experts suggest that managers of mission-critical IT centers simply need to harden existing facilities, other observers say they need to move the centers to higher ground and a third group says both strategies are needed.

But one thing is certain, experts say: Few IT organizations -- even those that suffered or narrowly escaped damage from recent major storms -- are thinking long-term. Most IT leaders are, if anything, taking the path of least resistance and expense.

For instance, the data center response to Superstorm Sandy, on the East Coast at least, "is nothing more than hardening existing centers," says Peter Sacco, founder and president of PTS Data Center Solutions, a data center design and consultancy firm in Franklin Lakes, N.J. On the other hand, he says, because computers are networked, that deemphasizes "the importance of any single data center."

Internap, a colocation vendor, is strengthening its most at-risk facilities, including the building at 75 Broad Street in downtown Manhattan that flooded after Sandy hit. During the storm, fuel pumps shut down and Internap switched to a 1,200-gallon reserve fuel tank on a higher floor to keep servers running.

"No one expected Sandy to become as catastrophic as it was," says Steve Orchard, Internap's senior vice president of development and operations. "With Hurricane Irene the previous year, we're seeing a trend that's a little alarming."

The company has announced it's building a new data center in Secaucus, N.J. -- outside the flood plain. "We take climate change very seriously, and it does factor into our new site selection," Orchard says.

Hurricanes Katrina and Rita, followed by Gustav and Ike (2008) and Isaac (2012), slammed into the Gulf Coast with such ferocity that IT executives at Entergy, a $10 billion electrical power company with 14,000 employees, abandoned the idea of a single data center in New Orleans and went back to the drawing board.

Before Katrina hit in 2005, the power company, which provides both nuclear and fossil fuel electricity to 2.8 million customers, had its corporate headquarters in New Orleans and one data center in Gretna, La., just west of downtown New Orleans across the Mississippi River.

"We knew the data center was in the storm's way, and we made a decision after that event to move the data center because we were holding our breath," says Jill Israel, Entergy's CIO. "We didn't have flooding in the immediate area of our data center," she explains. "But there was no power, our lines were down and we had to run on our generator and keep topping it off. "

By winter 2006, the company decided to create two mirror-image data centers in Jackson, Miss., and Little Rock, Ark. In Little Rock, the company retrofitted an old library with sturdy brick walls, moving hardware and critical applications from New Orleans, piece-by-piece, to the backup facility by 2008.

Finally, by 2010, Entergy had completed its brand-new $30 million Jackson, Miss., data center. The company load-balances several systems including email between the two centers, Israel says. "Moving applications from New Orleans involved quite a choreography plan. Subsequent to Katrina we've had [major] storm events, including ice storms in Little Rock. But I no longer have to hold my breath," she says.

The company holds drills for hurricanes and storms every year "to get better and better at responding," Israel says. "One of the things we quickly recognized was how effective a dispersed workforce can be. Our employees can do a lot more things from remote locations and that has served us very, very well."

Not everyone seems to have absorbed the "take heed" message. IT shops in both Europe and America's Northeast seem to cling to the idea that superstorms are non-repeatable freaks of nature. In some cases, even among those affected by major storms, vigilance plays a chess game with artful forgetfulness as managers choose other IT priorities.

Story continues on next page

In Florida, replication is key

Flooding, high winds and hurricanes are yearly occurrences in Florida, and data center backup has become key to local government continuing to function during disasters. Longboat Key is a barrier island with a winter population of 7,000 people that swells to 22,000 in the summer. Located on the west coast between Tampa and Fort Myers, the town decided to replicate critical data services in a secure mainland facility in 2004.

"Florida was just inundated with hurricanes that year, so everyone fortified their disaster recovery plans and efforts," says Kathi Pletzke, Longboat Key's IT director. "We had to assure that if a disaster were to occur, such as a hurricane, we'd need to set up shop fairly quickly and start serving the needs of citizens."

Longboat Key falls geographically into both Sarasota and Manatee counties. Tom Warner, Sarasota County's disaster recovery administrator, began conversations in 2004 with IT leaders in several western counties throughout the state. The result: Warner co-founded the Florida Technology Disaster Recovery Consortium, with Kevin Kryzda, CIO of Martin County, to help one another with replication throughout the region.

"The thought process was: How can we help each other because we are in Hurricane Alley, and what can we put in place long before hurricane season starts to ensure critical applications can be safe?" Warner explains. They looked at data centers to see which were hardened, which category of hurricane they could withstand and if they had additional floor space to house the counties' servers in their location.

With Sarasota County in the lead, the group eventually brokered a deal with Verizon to secure rack and server space for several counties in one of Verizon's switching centers, a secure brick building with no windows located near a major highway and around 10 miles inland from Longboat Key.

The town has replicated its servers at Suncoast Technology Center -- the former Verizon switching center -- and "though we don't have everything on it, as budgets allow, we've replicated the critical applications, including a mirror image of our production server, payroll, general ledger and public safety data," Pletzke says.

Because there are a lot of elderly people in Florida, "we wanted to replicate our website in [adjoining] Martin County so that in the event of disaster, people could check up on their parents in Sarasota County and see that they are well," Warner says. The consortium has also boosted cooperation among many counties to ensure that IT managers can get emergency access to guest servers and generators to keep computers running smoothly.

-- Arielle Emmett

Ignorance isn't bliss

"What was perceived as a safe area before may not be now," says Rakesh Kumar, a Gartner vice president who specializes in data center and infrastructure issues. In Europe, especially, he cites freezing temperatures, coastal flooding and other unpredictable weather events. In Asia, tsunamis are a concern. "Until we have a major data outage, though, most clients are not calculating for risk or change; they're turning a blind eye to it," Kumar says.

Many of his European and U.S. clients praise the idea of doing thorough risk assessments and thinking proactively, long-term, he says. At least in theory.

But even now, six months after Sandy, most East Coast-based companies aren't being proactive about repositioning their data centers, experts say. "They're expanding in the same locations; they're not even thinking about moving," says Neil Sheehan, a data center architect and principal of Chicago-based Sheehan Partners Ltd.

In fact, he says, "we are looking for expansion for our clients in New Jersey right near the coast, [near] sites that flooded." Sheehan says with proper surveys of 500-year-flood levels, data center architects can determine the ideal height of first floors, so that flooding, if it occurs, happens in parking lots and not in data centers.

Some are paying more than lip service. In downtown Manhattan, at 140 West Street, a Verizon switching center felt the full force of Sandy's stormwater. Five sub-basement levels, including a Verizon cable vault, went underwater. Technicians struggled to mount emergency generators and pump water out through elevator shafts.

Since then, Verizon has had to extract 150 tons of damaged copper cable from lower Manhattan streets, its central office, headquarters and customer sites, replacing virtually all of it with weather-proof fiber cable protected in conduit. "If you take a fiber optic cable and lay it in your bathtub it probably will still work; fiber is submersible," says Chris Kimm, vice president of global customer assurance for Verizon Enterprise Solutions.

By replacing electrical infrastructure and bringing it up to higher floors, the carrier was able to get its own central office back online in about a week.

Some customers are following suit -- moving critical infrastructure to floors that once housed rentable office space. And some are deploying new services, Kimm says, including mobile wireless, wireless IP and cloud computing solutions to allow their employees to work remotely. Others are rerouting their telecom and data networks. "We even have some buildings that landlords have to redo, converting them from business locations to residences and deploying wireless services."

Even given all the damage, however, Verizon isn't considering moving its switching centers to a less flood-prone location. Instead, Kimm says, "We're armoring the buildings; we've done an evaluation of what all the risks are," he says. "We haven't gotten final solutions, but the start of the next hurricane season [is a few months off]; you've got until mid-summer before you have a significant risk of a future event."

Lessons learned?

The impact of climate change and storms like Katrina and Sandy remains difficult to calculate. Not even climatologists can predict the frequency of extreme weather events as ocean levels and temperatures rise. But in the U.S., locations such as Manhattan, Long Island, New Jersey, Miami, Virginia Beach, Boston, Washington, D.C., and even Seattle and San Diego are expected to see increased coastal flooding.

"I think it's absolutely compelling to look at the impact of recent storms, and also to look at statistics that show there have been more natural [severe] weather events, whether that's related to global climate shifts or other factors," says Jim Grogan, a business continuity and resilience specialist at 451 Research.

"Every single event, though, leaves lessons to be learned," he says. "Lessons come from the stories of the creative and innovative things data center operators did to keep their centers" going even in the worst conditions.

Though Sandy may have been a wake-up call for major data centers in the New York area to take some steps to harden facilities, it remains unclear how many will act on a longer-term solution -- moving out of the city entirely, for example, or developing redundant and geographically separate facilities, or opting for third-party disaster recovery and cloud solutions.

New thinking is happening among some of the larger IT organizations. "We're definitely seeing companies looking at alternative locations for data center operations," says 451's Grogan. "We have spoken to multitenant data center operators in Atlanta, Virginia and other locations, and we're seeing an increase of interest from customers in the Northeast."

Many businesses that can afford a disaster recovery solution are looking now toward secured multitenant providers outside major cities. Cervalis, for example, a company with more than 200 large corporate clients including global banks and software and testing companies, has hundreds of thousands of square feet of multitenant space in upstate New York, Passaic County, N.J., and Fairfield County, Conn., all safely tucked away from floodplains.

The company acts as a primary or secondary data center, providing secured cabinets and redundant power, fully loaded (20 to 40 servers per cabinet), for about $1,500 per month per cabinet. That price is still cheaper for companies than building a redundant data facility on their own, insists Zack Margolis, a Cervalis vice president of sales, marketing and business development.

But not all businesses can afford this level of protection. Small businesses might realize the importance of data backups, but most will have to avail themselves of less expensive forms of resilience, such as backup tapes mounted at small disaster recovery facilities or virtualized clouds.

Moreover, many businesses still cling to major cities as the center of the universe. "Downtown Manhattan will still be in high demand because it is an interconnection hub to the United States," says 451's Michael Levy. Further, financial companies still want to "touch their data" and have it near the center of trading action without fear of latency, even though fiber-connected facilities outside the city can do the same job.

1 2 Page
Insider: How the basic tech behind the Internet works
Join the discussion
Be the first to comment on this article. Our Commenting Policies