U.S. intelligence agencies embrace OpenStack

The U.S. National Security Agency has been “transformed” since implementing OpenStack, and now the agency plans to open its experiences to all 16 agencies that make up the U.S. intelligence community.

“Over the next few months we’ll work with the larger intelligence community to roll out systems across the community,” said Nathanael Burton, a computer scientist with NSA, during a keynote at the OpenStack Summit in Portland, Oregon. “Hopefully we’ll be giving access to our OpenStack system to the rest of the [intelligence community] so they can leverage the same efficiencies.”

It wasn’t immediately clear if other agencies have committed to building their own OpenStack clouds or if they plan to use the NSA’s.

But government organizations are notoriously difficult to crack so the NSA’s successful implementation could open the door to broad usage of the cloud technology in the U.S. federal government.

It’s also good news for OpenStack users. Since the NSA has very strong security requirements, it developed a number of systems for securing APIs and guest OSes and putting SSL “everywhere,” Burton said. “I hope in the future to take what we learned from securing OpenStack and release that back to the community,” he said.

Because of the sensitive nature of the NSA’s work, Burton said he couldn’t reveal specifics like the number of servers, storage capacity or apps used in the OpenStack cloud. But he had plenty to share about how the group implemented the cloud and how it changed the way the agency works.

The NSA set out to improve upon the IaaS system it had in place. “It was manually intensive,” Burton said. He said the system included “stovepipes of excellence” that worked well individually but not together. Developers had to fill out paperwork, get approval, and repeat the process, sometimes for weeks or months before getting started. By the time they got approval, developers sometimes wondered why bother moving forward.

Burton attended the OpenStack Diablo working summit in 2011, returned to the office, “stole a rack” to avoid the paperwork process, and had a trial cloud up and running in two weeks. “It was a bit rough around the edges but we got it working,” he said.

Initially, there were “tens” of users in the lab environment, meaning they couldn’t access any data in the other system. But it proved what was possible. Users no longer had to submit requests and wait for approval.

So Burton launched a second OpenStack system on a half a rack, which has since tripled in size. Users were given access to mission data so they could develop projects with real potential use.

In six months, it had hundreds of users. “People could try out new ideas and whether or not they succeed isn’t important. It was giving people incentive to try things out,” he said.

Still, he wanted to make it a production cloud for the entire workforce at the agency and to do so had to solve some development and hosting problems, including changing the “stovepipes of excellence” model. Now, he’s at the point where he can take bare metal racks and servers and have a functioning OpenStacck cluster in 20 minutes, he said.

Without Burton’s group publicizing the existence of the OpenStack cloud, “we had an epidemic,” he said. Thousands of users began running production workloads on the new cloud.

He’s able to manage the cloud with a team of 15 people, far fewer than in the stovepipe model, he said.

Users are now sharing developments with each other. For instance, some have built recipes for deploying standard app stacks for things like Django, so others don’t have to reinvent the wheel each time they want to deploy it. “It’s led to better collaboration between developers,” he said.

There’s no word on how Burton’s comments about spreading OpenStack across the intelligence agencies impacts reports that the CIA was planning to hire Amazon to help it build a cloud.

Other hot topics at the OpenStack Summit have been worries that a lack of maturity of the technology is slowing down enterprise adoption and more talk of businesses leaving Amazon Web Services.

Read more of Nancy Gohring's "To the Cloud" blog and follow the latest IT news at ITworld. Follow Nancy on Twitter at @ngohring. For the latest IT news, analysis and how-tos, follow ITworld on Twitter and Facebook.