Web hosting outage a teachable moment on backup power

The Web hosting company The Planet.com Internet Services Inc. tests its backup generators monthly and some employees ask if that's really necessary, said manager Urvish Vashi. The blackout in San Francisco Tuesday explains why.

Among the 40,000 customers affected by a nearly two-hour electrical blackout in San Francisco was 365 Main Inc., a Web hosting company whose clients include Sun.com, Yelp.com and Craigslist.com. Their Web sites were among several that were unavailable Tuesday when the local electric utility, Pacific Gas & Electric Co. (PG&E), suffered an outage that cut power to the southern part of the city, including the South of Market area, or SOMA, which is home to several technology companies.

The outage served as a reminder of the importance of backup power systems to keep Web sites, and the businesses behind them, running.

More information was provided Wednesday by 365 Main about what happened. An electrical surge caused PG&E transmission breakers to open, interrupting power to the 227,000-foot data center. That event automatically signaled backup diesel generators to power up, but some didn't, the company said in a prepared statement.

"On-site facility engineers responded and manually started affected generators, allowing stable power to be restored ... across the entire facility," 365 Main stated. It was without power for about 45 minutes.

The company did not specify how many of the generators failed to start or why, but estimated that the interruption affected between 20 percent and 40 percent of its customers. It remained on generator power Wednesday, saying it first wanted assurances from PG&E that its power supply was stable.

One page on 365 Main's Web site contains video tours of its facility, which the company acquired from the Web hosting company AboveNet.com after its parent company, Metromedia Fiber Network Inc., went through financial reorganization in U.S. Bankruptcy Court in 2003.

Jean Paul Balajadia, vice president of operations, explains on the video that 365 Main operates ten 3,000-horsepower diesel generators that each feed a 12,000-pound flywheel that connects to a 2.1 megawatt electrical generator that powers the data center. "During a power outage. ... the diesel engine is called to fire up and starts up in about 2.1 seconds," Balajadia explains.

Web hosting companies usually provide service level agreements (SLAs) to customers specifying the percentage of time the data center promises to be in operation. The Planet.com, which has data centers in Houston and Dallas, offers one SLA that offers 99.99 percent availability and a more expensive one that guarantees 100 percent. If the hosting company falls short of its SLA, the customer is credited up to the full monthly fee for the hosting service, said Vashi. The Planet has suffered power or network outages but none that resulted in a "customer facing outage."

The Planet.com took extra precautions in September 2005 when Hurricane Rita threatened Houston, ordering extra truckloads of diesel fuel in the event a power outage brought its Houston center down, said Yvonne Donaldson, a company spokeswoman. The Planet.com also has battery backup to keep the servers running while the generators boot up.

365 Main, a San Francisco company, doesn't use battery backup because it keeps its backup generators running continuously, said Miles Kelly, vice president of marketing, in an e-mail message.

The company, which operates five other centers and is developing a sixth, is chastened.

"We will apply knowledge gained in this investigation to all 365 Main facilities to help prevent this type of incident from happening again," the company said in a prepared statement.