October 03, 2002, 2:15 PM — WorldCom Inc.'s UUNet Internet backbone provider is experiencing service interruptions throughout the U.S., causing a ripple effect that slowed down many Internet connections Thursday.
A brief message on WorldCom's Internet Technical Support Center hotline advises customers that the company is experiencing interruptions and working on correcting the problem.
WorldCom has restored service to some customers and is working with its vendors to determine the outage's cause, said spokeswoman Jennifer Baker, early in the east-coast afternoon. A 3 p.m. ET update to WorldCom's hotline still warns of network problems, and several Internet monitoring services continue to show significant back-ups between major ISPs (Internet service providers) and their connections to UUNet.
"I personally have never seen anything like this. This is a bad event that's occurring," said Tom Ohlsson, vice president of marketing and business development for monitoring firm Matrix NetSystems Inc., in Austin.
Matrix NetSystems first recorded UUNet problems at 7 a.m. ET Thursday, according to Ohlsson. WorldCom's Baker said the problems began around 8 a.m.
The problem occurred suddenly, Ohlsson said: One minute, UUNet was fine; the next, massive lag hit the system.
"What we believe happened is that overnight, WorldCom network engineers loaded new operating system software onto most or many of their border routers," he said.
Border routers connect one system to another, in this case linking UUNet to other regional and national ISPs. UUNet is an Internet backbone, carrying traffic originating from a number of ISPs; Matrix NetSystems estimates that 65 percent of the world's Internet traffic flows over UUNet.
One indicator tracked by Matrix NetSystems is packet loss, a measure of how much data is being lost in transmission. Yesterday, UUNet's highest packet loss for the entire day was 0.04 percent. At 7 a.m. ET, UUNet's packet loss rate suddenly shot up from around 0 to 22 percent -- a change consistent with Matrix NetSystems' hypothesis that new, buggy software was abruptly activated, Ohlsson said.
Baker declined to comment on whether new software has been installed, reiterating that WorldCom continues to work with its technicians and vendors to identify the outage's cause.
Several online sites that monitor Internet traffic patters are charting the ongoing problems. Opnix Inc.'s Internet Traffic Report, at http://www.internettrafficreport.com/, shows marked packet loss problems and response-time lags worldwide. Keynote Systems Inc.'s Internet Health Report, at http://www.internetpulse.net/, is reporting critical latency in the connections between UUNet and several other major ISPs, including AT&T Corp. and Sprint Corp.
Sprint's connections to other ISPs are fine, but its UUNet connection has been up and down since around 8 or 8:30 a.m. ET, spokesman Charles Fleckenstein said. When Sprint customers try to reach Web sites on UUNet's network, they're hitting lag and outages, he said.
Sprint's network engineers have been in touch with their counterparts at WorldCom, but solving the problem is in WorldCom's hands, Fleckenstein said. "It's their network," he said.
