Business Continuity Challenges in a Hurricane Zone

By Joan Goodchild, CSO |  Security, business continuity, disaster recovery

The corporate headquarters building for OSI Restaurant Partners is a mere 800 feet from the end of a runway at Tampa International Airport. But according to OSI Chief Information Officer Dusty Williams, that's the least of their concerns.

OSI, the company that owns popular restaurant-chain brands such as Outback Steakhouse, Roy's and Carraba's Italian Grill, is smack dab in the eye of the storm zone, in hurricane country. Their 750-person operation in Tampa includes all back office functions, including the financial, legal and real estate divisions. If a hurricane strikes and the building is impacted, the amount of sensitive data that is at stake is immeasurable.

"We're in an A zone as far as flooding is concerned. You don't really want your data center here."

The 2008 Atlantic hurricane season produced a record number of consecutive storms, according to National Oceanic and Atmospheric Administration. The season saw a total of 16 named storms. With water temperatures rising due to climate change, many meteorological experts predict even tougher seasons to come. For companies in a hurricane zone, business continuity and disaster recovery preps need to be in place now, and not when the storm clouds begin churning.

It is that kind of thinking that inspired Williams to find a new home for the data center. In 2003, the main data center in headquarters had no back up power and a business continuity/disaster recovery plan was a vague notion. Williams got initial approval to move OSI's data center to an off-site facility hosted by backup and storage service provider Qwest.

"Typically when we talk BC/DR, it's always around hurricanes. The plan was to move the data center locally to a Qwest facility," said Williams. "The building itself is a category 3 or 4 that is built to sustain hurricane damage and has back up and battery power that we don't have in the headquarters facility."

Within months, the plan was put to the test. Florida experienced a severe hurricane season in 2004. Williams said Hurricane Charley illuminated the fact that they had made the right decision to move data off-site.

"On a Thursday night at 5 o'clock, officials told us they would be shutting power down to the grid we are on. So, if we had not outsourced the data center, we would have been dead in the water. "

Williams said the entire summer of '04 was spent preparing for hurricanes. At least four blew through the area of varying intensity. While no major damage was sustained, when the season was over, it became clear that the BC/DR plan needed to include more than just one off-site data facility. OSI now has a second cyber center in Chicago that includes all critical systems. The company has more than 1200 restaurants around the country. The Chicago center would allow OSI and its restaurants to have operations back up and running within a few hours if the Florida off-site facility went down, according to Williams' estimate.

OSI's BC/DR plan is tested regularly to ensure connectivity to restaurants is maintained. Williams says he tests by bringing the main data center down and bringing the Chicago facility online.

Outsourcing the data center is crucial to any business with a natural disaster risk, according to Iain Hardcastle, senior consultant with professional services firm Deloitte & Touche at their operations in Bermuda. On the small island where his company operates, there is only one power supply. The local office, which stores all data on a SAN, also replicates the information at a local data hosting center.

"The accounting side of our business is managing trust funds and looking after accounts for many name-plate companies. They can be absolutely multimillion-dollar, global clients. They don't care if we have a bit of a weather problem down here."

"Buns on seats" preparations
The data is only one part of the picture when it comes to business continuity in a natural disaster-prone area. If a facility goes down because of power failure or flooding, many organizations need a physical location to place their staff so operations can continue. Deloitte has what Hardcastle refers to as a "buns on seats" office off-island. So, too, does OSI. OSI maintains a comprehensive facility in Atlanta, which they have had to use at least twice in the last 4 years.

"Once we declare a disaster, we have 50 cubes available there," said Williams. "But we have to go up and make sure everything is up and running and ready. So we have people, from an IT perspective, head up 72 hours out ahead of any storm in private aircrafts to make sure everything is ready to go."

Sometimes it isn't just humans that need to be relocated. One year, according to Williams, OSI tried to send a check printer up in a plane so vendor checks could continue to be cut. Unfortunately, the machine didn't fit through the door of the aircraft. The check printer was delivered to Atlanta by van instead.

The process of relocating people, and sometimes equipment is time consuming, labor intensive and costly. The company even has contracting companies on standby for employees that may need assistance with boarding up houses before they depart. As complicated as it all sounds, Williams says, thankfully, most of it can be planned.

"With hurricanes, you have a distinct advantage over an earthquake or a tornado. You really don't know when they will strike."

Can you ever be completely prepared?
Even the most comprehensive BC/DR plan isn't without some risk, according to Hardcastle, who calls the Deloitte BC/DR plan a "continuously evolving process."

Williams admits he is still troubled at the prospect of keeping track of personnel in a worst case scenario.

"I don't worry as much abut the technical side of it as a do the operations/people side of it. How do you find people?" he said.

OSI says disaster plans are also considered regionally for all of its 1200-plus restaurants and each have special numbers set up so people can dial-in and alert the company as to where they are.

"But you worry about how long that will take if cell service, phone service, is down" said Williams.

And despite the plans put in place at the headquarters building, there will still inevitably be some loss if the facility itself is damaged in high winds or flood waters, said Williams.

"Sometimes people have paper on their desk that they haven't put into a system yet. In those cases you need to ensure you have connections with vendors to ask them "How can we get your invoice back in here and get you paid?"

Join us:
Facebook

Twitter

Pinterest

Tumblr

LinkedIn

Google+

Spotlight on ...
Online Training

    Upgrade your skills and earn higher pay

    Readers to share their best tips for maximizing training dollars and getting the most out self-directed learning. Here’s what they said.

     

    Learn more

Answers - Powered by ITworld

ITworld Answers helps you solve problems and share expertise. Ask a question or take a crack at answering the new questions below.

Ask a Question
randomness