A year after distributed denial-of-service attacks blasted the likes of Yahoo Inc., eBay Inc. and E-Trade Group Inc., no one has found an easy way to defend against a flood of unwanted IP packets.
In fact, everyone's still pretty much in the dark -- literally, in one case -- when it comes to finding a silver bullet.
A recent meeting of the DDoS Working Group, a forum organized last year to plot network defenses, was conducted solely by the light of laptops after KPMG International's Silicon Valley office was visited by one of California's rolling blackouts. In the ghostly glow could be discerned John Zent, manager of risk management for Yahoo, and Allen Yousefi, information security officer at eBay, along with representatives from security vendors eager to woo these top e-commerce firms.
The talk was no brighter than the lighting. According to several attendees, Yahoo and eBay are more than just dismayed by the slow pace of finding technical defenses to denial-of-service attacks and the even more nefarious distributed denial-of-service attacks, which let an individual launch IP attack streams from hundreds, or even thousands, of compromised computers.
Web site operators are frustrated by the apparent inability of ISPs and Web-hosting providers to quickly filter out denial-of-service attack traffic when it pours into their routers and servers. Whether a low-grade nuisance or the kind of multibarreled assault that upended Microsoft's sites for three days recently, this "bad" traffic is eating up bandwidth and at times blocking legitimate traffic to the most prominent e-commerce sites.
"People are getting a little radical about it," said one attendee. For companies such as Yahoo and eBay, "it's a service-level agreement [SLA] issue with the ISPs and collocation providers." He predicted this year will see lawyers battling over whether distributed denial-of-service traffic should have to be filtered out to satisfy SLAs.
Despite the gloom, there are many efforts under way to cope with all manner of denial-ofservice threats . . . and rays of hope are visible:
- Service and software providers have united to share information and forge common defenses.
- Promising security start-ups focusing on the problem are attracting big-name backers.
- Law enforcement groups -- working with the network industry and its customers -- are nailing the bad guys.
The DDoS Working Group is doing what it can to spur cooperation among ISPs. The group plans to publish recommendations for automated distributed denial-of-service defenses by the end of March.
"There are political issues and technical issues," says Tom Clare, a product manager for Check Point Software Technologies Ltd. and DDoS Working Group member.
The document is expected to define a common intrusion-detection method for collecting and measuring the percentage of bandwidth being consumed and a flow tag to identify traffic and other Layer 2 data collected from the packets. A firewall or other network device that implemented the DDoS Working Group specification would be able to report the start of an attack to the ISP, and other ISPs using compatible equipment would be able to share the information.
But it's uncertain whether ISPs can interact smoothly even if equipment makers support a common security specification, which may leave this as yet another security proposal that never got off the drawing board.
ISPs in the middle
This much is clear: ISPs play the critical role in the distributed denial-of-service endgame against attackers, who are heavily armed with denial-of-service "malware," software posted at hacker sites for free use. And most of the intrusion-detection analysis and filtering that ISPs do is manual and difficult.
"We can't be held responsible for attacks on our customers," says Amir Moujtahed, director of systems engineering and corporate security at Epoch Internet, a Costa Mesa, Calif., ISP. "But if customers give us the IP addresses [of the source], we will block them." Epoch has intrusion-detection equipment from NFR Security on its external and internal networks, and Epoch engineers watch the logs closely for evidence of attack signatures. But it's a labor-intensive process.
Moujtahed says ISPs are trying to do their part by installing antispoofing filters and cooperating with competitors through informal agreements hashed out in the ISP Service Consortium, which meets monthly.
"This is all part of the lesson learned after what happened last year," Moujtahed says. "ISPs like [Genuity Corp.], UUNET (Technologies Inc.) and AOL (America Online Inc.) compete, but we are working together on this."
It's small comfort to the high-tech industry that the 16-year-old perpetrator of last February's incidents, a Canadian hacker nicknamed Mafiaboy, last month pled guilty to single-handedly attacking Amazon. com Inc., eBay, Yahoo, Charles Schwab & Co., CNN and eTrade, among others.
Mafiaboy carried out his distributed denial-of-service spree using attack tools available on the Internet that let him launch a remotely coordinated blitz of IP packets from servers compromised by agent attack "zombies." Mafiaboy awaits sentencing, but it's expected he won't get much more than two years in a juvenile detention center.
Those attacks forced most of the victimized e-commerce sites offline for about three hours. In the heat of battle to block the blitz of IP packets, ISPs did what they could through filtering bad traffic and claimed victory when it ended. But security experts familiar with what occurred agree that this filtering accomplished little and that relief came because Mafiaboy simply stopped his attacks after three-hour intervals.
"The attacks happened Monday through Wed-nesday, and those guys were still working Friday and Saturday to figure out what happened," says Frank Huerta, CEO of Recourse Technologies Inc., which makes security gear to detect and trace denial-of-service attacks.
Like many experts, Huerta says the work ISPs did manually filtering bad traffic didn't stop Mafiaboy's attacks. And though law enforcement officials did extensive work bringing him to justice, one reason they succeeded was that he bragged about his exploits in an Internet chat room.
Microsoft two weeks ago became the latest high-profile victim of a distributed denial-of-service attack, though no one seems to be bragging about causing it. The software giant lost MSN.com, Carpoint.com, Expedia.com and other Web properties for a day, hours or minutes over the course of a week.
Microsoft declined to explain its response to the attacks, other than to say it was working with the U.S. Federal Bureau of Investigation. However, CIO Rick Devenuti acknowledges that Microsoft "accepts full responsibility" for the inconvenience to its Web users. He says the company hadn't applied "sufficient self-defense" by using third-party products at the front end of its core network.
There are stopgap measures that Web sites can take to shore up defenses, such as using as many load-balancing and high-speed pipes as they can, as well as intrusion-detection systems that can indicate suspicious activity is suddenly on the radar screen.
And that is better than nothing. Fidelity Investments and Bear Stearns reportedly deployed Top Layer Networks Inc.'s AppSwitch with its intrusion-detection features after last February's attacks on e-commerce sites.
Finding a cure
Overall, there's a more sober-minded assessment of the problem among vendors than a year ago.
Cisco Systems Inc. last February claimed that making use of ingress filtering in routers, a technology described in IETF draft RFC 2267plus, would stop denial-of-service attacks. But the router manufacturer has abandoned that stance.
"There is no silver bullet for a [denial-of-service] attack," says Lance Hayden, a manager with Cisco's consulting services team in Austin, Texas. But Cisco and a number of venture capital firms are investing in start-ups that are promising to develop comprehensive defense systems for distributed denial-of-service attacks. Another start-up, Arbor Networks, is also striving to find a cure.
So, too, are established security vendors, including Internet Security Systems (ISS). Allen Wilson, director of emerging technologies at ISS and a DDoS Working Group member, says tracing this type of attack remains "very manually intensive and time-consuming. For ISPs, it's one hop at a time, and you need to get hold of people and let them know that your network is attacking theirs."
ISS claims to be developing technologies that depend on what it calls "the moving target defense." The idea is that if an attack is launched at a Web site, the victim and ISP work together to identify the source and then create a "black hole for the IP address," Wilson says. "You drop the packets but don't kill the connection, which helps trace back the attackers."
At the same time, you create a temporary IP address for your site that gets broadcast out to enable legitimate traffic to still find you.
Quantifying the denial-of-service problem is not easy. Whenever a Web outage occurs, security experts always suspect denial of service, even if the business blames internal screw-ups. Online auction vendor eBay has suffered several Web outages in recent months that many security experts suspect were denial-of-service attacks, something eBay vehemently denies. However, it was clearly a denial-of-service attack that disabled much of the Undernet, part of the Internet Relay Chat network, in early January.
After last February's attacks, the Clinton administration asked the IT industry what it could do to help combat what everyone suddenly realized was a dangerous situation on the 'Net.
It took 11 months to come forward with a plan, but 19 high-tech corporations recently formed an organization called IT Information Sharing and Analysis Center (IT-ISAC), which will run a so-called "virtual center" to share information about denial-of-service attacks and software vulnerabilities in general. Founding members are paying almost $1 million for the privilege, although general membership fees, which won't include access to all the information, drop as low as $5,000.
The organization's database of shared information, which will be managed by ISS, is intended to help solve security problems, so vendors accessing this sensitive information have agreed not to use it as a marketing weapon.
Those who expected ISPs to roll out new technologies or services to help stop these attacks in the past 12 months have surely been disappointed. ISPs are essentially using the same spot-filtering and monitoring techniques today as a year ago. Nevertheless, ISPs claim heightened awareness and vigorous monitoring have helped reduce damage.
"We regularly see attacks, but nothing at the level of last year's on multiple, highly visible customers," says Kelly Cooper, security engineer at Genuity. "If we were to offer filtering and monitoring services to our customers for an extra charge, that would sort of be like blackmailing them."
Genuity expects new capabilities from router and switch vendors that will integrate IP address filtering directly into the operating system of the device. One of the most common reasons why ISPs are not setting up IP address filtering is because it can slow the network. However, if filtering is integrated into network devices, performance should not be hurt, Cooper says.
Vint Cerf, senior vice president of Internet architecture and technology at WorldCom Inc., says that standard load-balancing and content-distribution techniques that many Web-hosting service providers use reduce the negative impact of these attacks.
"Load sharing across multiple servers helps reduce the impact of classic [distributed denial-of-service] attacks because there are multiple versions of a Web site operating across the Internet," Cerf says. In addition to distributing legitimate traffic, load balancing and caching distribute rogue distributed denial-of-service packets so one server is not crumbling under the weight of an attack.
ISPs also see hope in specifications being developed by the Internet Engineering Task Force. I-Trace is one preliminary technology that will allow ISPs to quickly find where a distributed denial-of-service attack originates. Once the ISP recognizes the source of an attack it can immediately set up a filter.
But this technology is very much in the early stages of development. All in all, it certainly seems like the industry will experience at least another year of being in the dark on distributed denial of service.
This story, "DDoS: One year after" was originally published by Network World.