YouTube outage underscores big Internet problem

by Robert McMillan

February 26, 2008 —

Sunday's inadvertent disruption of Google's YouTube video service underscores
a flaw in the Internet's design that could some day lead to a serious security
problem, according to networking experts.

The issue lies in the way Internet Service Providers (ISPs) share Border Gateway
Protocol (BGP) routing information. BGP is the standard protocol used by routers
to find computers on the Internet, but there is a lot of BGP routing data available.
To simplify things, ISPs share this kind of information among each other.

And that can cause problems when one ISP shares bad data with the rest of the
Internet.

That's what happened with YouTube this weekend, according to sources familiar
with the situation. BGP data intended to block access to YouTube within Pakistan
was accidentally broadcast to other service providers, causing a widespread
YouTube outage.

The chain of events that led to YouTube's partial black-out was kicked off
Friday when the Pakistan Telecommunication Authority (PTA) ordered
the country's ISPs to block access to YouTube because of an alleged anti-Islamic
video that was hosted on the site.

According to published reports, the clip was from a film made by Geert Wilders,
[cq] a Dutch politician who has been critical of Islam. Wilders is hoping to
air a 15 minute anti-Islam film, called
Fitna on Dutch television in March.

ISPs in Pakistan were able to block YouTube by creating BGP data that redirected
routers looking for YouTube.com's servers to nonexistent network destinations.
But that data was accidentally shared with Hong Kong's PCCW, who in turn shared
it with other ISPs throughout the Internet.

In San Francisco, David Ulevitch first
noticed the problem Saturday morning. "I was trying to watch cats falling
off roofs... and I couldn't get to YouTube," he said. Ulevitch, who runs
an Internet infrastructure company called OpenDNS, was soon able to connect
with engineers at Google, who also experienced similar problems, he said. "They
were like, 'Holy crap, we can't get to YouTube either.'"

Because Pakistan's BGP traffic was offering very precise routes to what it
claimed were YouTube's Internet servers, routers took it to be more accurate
than YouTube's own information about itself.

Larger service providers typically validate BGP data from their customers to
make sure that the routing information is accurate, but in this case, PCCW apparently
did not do that, according to Ulevitch. When the Pakistani ISP sent the bad
data, PCCW ended up sharing it with other ISPs around the globe.

This kind of accidental denial of service attack has
happened before. In early 2006, for example, New York's Con Edison caused
data intended for a number of networks to be misrouted following
a similar mistake.

There wasn't anything that Google could have done to prevent the problem, said
Danny McPherson, [cq] chief research officer with Arbor Networks. "They
can't keep someone on the Internet from announcing their address space,"
he said. "It's a huge vulnerability."

By intentionally propagating bad BGP data, an attacker could knock a Web site
off the Internet or even redirect visitor's traffic to a malicious server, security
experts said.

Although there hasn't been a high profile example of criminals misusing the
BGP protocol to knock a Web site offline intentionally, it has been misused
by spammers to cover their tracks.

If criminals were able to send BGP information to a larger service provider
that didn't properly check its BGP data, they could cause serious problems,
McPherson said. "The reality is that if you wanted to cause global instability,
you simply compromise one of those people who have access to a BGP-speaking
router," he said.

Making BGP data more reliable isn't so easy either. Although secure versions
of BGP have been developed, it would take a major effort to adopt them and until
there's widespread concern over the current system, it is likely to continue.

Two parties were to blame for the YouTube fiasco, said a networking engineer
familiar with the YouTube situation, who asked not to be identified. First,
the Pakistani ISP should never have forwarded the bad BGP routing data to PCCW.
Second, PCCW should have checked to make sure that the ISP was talking about
its own domains before accepting the information.."One of the dirty secrets
about the Internet is a lot of it is still a handshake deal," he said.

IDG News Service