June 19, 2013, 8:17 PM — With more than a billion monthly active users, it's easy to imagine that most of the data travelling over Facebook's networks is delivering photos, status updates and "likes" to its end users, but that's far from the case.
The social network moves about 1,000 times as much data between the servers inside its data centers as it does from its servers out to end users, company executives said Wednesday. They talked about the challenges that this creates for Facebook and the network technologies it's developing to overcome them.
"Our traffic going from machine to machine far exceeds the traffic going from the machines out to our end users," said Jay Parikh, vice president of Infrastructure Engineering at Facebook, in an on-stage interview at the GigaOm Structure conference in San Francisco.
That's because of all the processing work Facebook does on the back end to figure out what information it needs to send to end users. The systems analyze data, rank results, and perform a myriad of other tasks to generate the pages Facebook delivers to users' smartphones and Web browsers.
The problem, as Parikh described it, is that Facebook is using network equipment and protocols that were designed for a different era: for the ISPs (Internet service providers) whose chief role was to provide connectivity for end users, rather than companies like Facebook that are delivering applications and content at massive scale.
One result is data bottlenecks. Facebook sees "flash floods, choke points, all types of scenarios," said Najam Ahmad, the director responsible for Facebook's network operations, who also talked at the Structure conference
"One of our services handles 2.5 billion operations per second; that's a lot of packets being sent over the wire," he said.
Another issue is that the network technologies aren't smart or flexible enough for the types of applications Facebook is running. "There's really no way to communicate between an application and the network today. The app just puts the packet on the [network] and hopes it gets to the other end," Ahmad said.
"We want to come up with technologies that give the app a better feel for what the network is doing -- where it has capacity, where there are problems ... where you can find a better path."
It's addressing the network challenges in a couple of ways. One is by designing new switching hardware through the Open Compute Project, a multi-company effort to build new types of equipment for Internet-scale data centers. OCP's initial goal for networks, announced in May, is to design a top-of-rack switch that's not tied to any one operating system.