June 26, 2012, 1:59 PM — One of the biggest users of Amazon's cloud services, Netflix has released an open source console called Asgard that it developed to manage its Amazon deployments.
A home-built console was necessary for Netflix, given the amount of work it conducts on Amazon Web Services (AWS).
"For everyday large-scale operations, the AWS Management Console has not yet met the needs of the Netflix cloud usage model, so we built Asgard instead," wrote Joe Sondow, the Netflix software engineer for engineering tools who led development of the application.
Amazon's own management tools have a number of shortcomings, Sondow said. For instance, Amazon offers no way to automate the workflow around deploying an application. It does not provide a log of recent administrative actions. And it doesn't provide many ways to integrate in-house engineering tools, such as Jenkins.
Sondow built the application to address these issues, using the Grails framework. Named after the home of the Norse god of thunder and lightning, Asgard allows administrators to manage their Amazon cloud deployments from two perspectives unavailable on Amazon's own management interfaces, that of applications and clusters.
For Netflix, running an application may require multiple Amazon services, including those for load balancing, auto-scaling, launching and security. Asgard provides a single view for managing these services, organized around each application. Asgard maintains an application registry in Amazon SimpleDB, which keeps track of the many resources supporting a single application. It can also show how long each instance of the application has been running and when it was created.
Asgard has a similar interface for managing clusters, one that lists the individual nodes of each cluster. The software can remove individual nodes, a practice called rollback, if they prove to be troublesome. It also can manage a rolling push, in which all the nodes of a cluster are replaced one by one, without disrupting the operation of the cluster as a whole.
Netflix' use of Amazon is expansive. Except for billing and a number of other functions, "We run all of Netflix out of EC2," explained Jeremy Edberg, the lead cloud reliability engineer at Netflix, during a talk at the Qcon developer conference last week in New York City. The company had to use a public cloud because "we couldn't build data centers fast enough," he said. Netflix has 25 million customers who stream movies and televisions shows from the service.
