A Web hosting company Wednesday cited a faulty controller as the reason its backup electrical system failed during a San Francisco power outage last week, dousing the Web sites of several customers.
Service was interrupted to 40 percent of 365 Main Inc.'s customers on July 24 when an electrical surge caused Pacific Gas & Electric Co., the local utility, to shut off power. Three of 10 backup generators failed to start at 365 Main as expected and the Web sites Craigslist.com, Red Envelope.com, Yelp.com, Technorati.com and others went down for about 45 minutes.
365 Main traced the generator failures to a weakness in a controller in back-up diesel generators. An incorrect setting on the device -- called a DDEC, for Detroit Diesel Electronic Controller -- was not allowing it to correctly reset its memory. Erroneous data left in the DDEC's memory subsequently caused the diesel generators to misfire or fail to start, the company said.
365 Main corrected the settings on all the generators at its San Francisco data center and at another center it operates in El Segundo, California, because both use the same model of generators from Hitec Power Protection Ltd. Three other 365 Main data centers use other Hitec models.
The power outage resulted in 365 Main breaking service level agreements it has with its client Web sites and 365 Main says, as a result, it will refund a portion of the monthly Web hosting fee the clients pay. Miles Kelly, vice president of marketing and strategy at 365 Main, declined to specify the amounts being refunded.
Even with last week's outage, the San Francisco data center has delivered 99.9942 percent uptime, but that's no consolation to clients whose Web sites were out of service, said Kelly.
"What we learned from this is that there remains a delta between any number of nines and perfection," he said.
