February 26, 2010, 1:07 PM — SAN JOSE -- Vendors have a lot more work to do to make Ethernet ready for large-scale data center duty.
According to users at the Ethernet Technology Summit here this week, Ethernet could be a lot greener and "fabric-friendly." This shows that recent efforts by vendors and standards organizations -- such as the Data Center Bridging work by the IEEE and Cisco's next-generation Nexus platforms -- to ruggedize and reduce the power consumption of Ethernet switches and routers for data center applications are still incomplete.
"The biggest, baddest switch you can buy today is still too small -- woefully too small," said Donn Lee, an engineer at Facebook. "We have to lash together huge arrays of 10G links to scale. Switches are not built for being in a fabric."
Facebook is the fourth largest Web site in the world after Google, Microsoft and Yahoo, Lee said. And it is still growing: The social networking site jumped from 50 million users to 400 million users in the past two years, he said.
But that growth has not come without some Ethernet pain points, Lee said. Facebook has warehouse-sized data centers running racks and racks of commodity PCs -- between 20 and 40 per rack -- all running quad core processors.
And with each PC equipped with a 1Gbps Ethernet NIC, there could be as much as 40Gbps coming out of each server cabinet.
"We have to use 10G and a lot of 10G" to connect to aggregation switches and then backhaul traffic into the core of a single 700,000 square-foot data center, Lee said. That's one of the reasons Facebook said it needs 100G and even Terabit Ethernet now.
"It's a pain point we see with Ethernet," Lee said. "How do we build an Ethernet network to support tens of thousands of servers and thousands of 40G [cabinets]? It's not easy because the switches aren't big enough."
Some of the bigger Ethernet switches on the market support 500-plus 10G ports, but only 25% or less of them at line rate. That still doesn't meet Facebook's needs.