Memcache also required a fair amount of expertise from the developers who built Facebook user-facing products, Venkataramani noted. If these developers did not understand all the nuances of the Memcache, their products could have data inconsistencies, bugs and performance issues.
The TAO caching layer is run on the servers by a collection of daemons, mostly written in C and C++. They route write requests, execute read requests and maintain consistency with other caching servers. TAO cache servers are one of two types: leaders or followers. Each leader cache is assigned to a single database shard, and is responsible for maintaining the consistency of the data between itself and the shard.
The leader cache periodically sends updates to the follower caches, which are the caches that users first hit when requesting data from Facebook. Facebook works on the principle of eventual consistency, in which data written to Facebook will be made available for access, though a few seconds may lapse before the data is written to all the database and the caches. Eventual consistency has long been a behavior associated with using a distributed database.
TAO offers a number of advantages for Facebook, Venkataramani said. First, it scales easily for traffic spikes, simply by adding more follower servers. It also is easy to upgrade with because it cleanly separates the caching layer from the persistent data storage layer, allowing the company to update and scale either one without affecting the other. The API also cleanly separates the product logic from the data access. As a result, "when building products, the product engineers just use the API to store and access data," Venkataramani said.