YouTube outage underscores big Internet problem

The issue lies in the way Internet Service Providers (ISPs) share Border Gateway Protocol (BGP) routing information. BGP is the standard protocol used by routers to find computers on the Internet, but there is a lot of BGP routing data available. To simplify things, ISPs share this kind of information among each other.

And that can cause problems when one ISP shares bad data with the rest of the Internet.

That's what happened with YouTube this weekend, according to sources familiar with the situation. BGP data intended to block access to YouTube within Pakistan was accidentally broadcast to other service providers, causing a widespread YouTube outage.

The chain of events that led to YouTube's partial black-out was kicked off Friday when the Pakistan Telecommunication Authority (PTA) ordered the country's ISPs to block access to YouTube because of an alleged anti-Islamic video that was hosted on the site.

According to published reports, the clip was from a film made by Geert Wilders, [cq] a Dutch politician who has been critical of Islam. Wilders is hoping to air a 15 minute anti-Islam film, called Fitna on Dutch television in March.

ISPs in Pakistan were able to block YouTube by creating BGP data that redirected routers looking for YouTube.com's servers to nonexistent network destinations. But that data was accidentally shared with Hong Kong's PCCW, who in turn shared it with other ISPs throughout the Internet.

In San Francisco, David Ulevitch first noticed the problem Saturday morning. "I was trying to watch cats falling off roofs... and I couldn't get to YouTube," he said. Ulevitch, who runs an Internet infrastructure company called OpenDNS, was soon able to connect with engineers at Google, who also experienced similar problems, he said. "They were like, 'Holy crap, we can't get to YouTube either.'"

Because Pakistan's BGP traffic was offering very precise routes to what it claimed were YouTube's Internet servers, routers took it to be more accurate than YouTube's own information about itself.

Larger service providers typically validate BGP data from their customers to make sure that the routing information is accurate, but in this case, PCCW apparently did not do that, according to Ulevitch. When the Pakistani ISP sent the bad data, PCCW ended up sharing it with other ISPs around the globe.

This kind of accidental denial of service attack has happened before. In early 2006, for example, New York's Con Edison caused data intended for a number of networks to be misrouted following a similar mistake.

There wasn't anything that Google could have done to prevent the problem, said Danny McPherson, [cq] chief research officer with Arbor Networks. "They can't keep someone on the Internet from announcing their address space," he said. "It's a huge vulnerability."

By intentionally propagating bad BGP data, an attacker could knock a Web site off the Internet or even redirect visitor's traffic to a malicious server, security experts said.

Although there hasn't been a high profile example of criminals misusing the BGP protocol to knock a Web site offline intentionally, it has been misused by spammers to cover their tracks.

If criminals were able to send BGP information to a larger service provider that didn't properly check its BGP data, they could cause serious problems, McPherson said. "The reality is that if you wanted to cause global instability, you simply compromise one of those people who have access to a BGP-speaking router," he said.

Making BGP data more reliable isn't so easy either. Although secure versions of BGP have been developed, it would take a major effort to adopt them and until there's widespread concern over the current system, it is likely to continue.

Two parties were to blame for the YouTube fiasco, said a networking engineer familiar with the YouTube situation, who asked not to be identified. First, the Pakistani ISP should never have forwarded the bad BGP routing data to PCCW. Second, PCCW should have checked to make sure that the ISP was talking about its own domains before accepting the information.."One of the dirty secrets about the Internet is a lot of it is still a handshake deal," he said.