Florida CIO takes election to the cloud
People in the Sunshine State stood in line for hours to exercise their franchise, in some cases six hours or more after the polls had officially closed. Numerous causes have been cited, from a shortened early voting period to a 10-page-long ballot with 11 state constitutional amendment questions to massive voter turnout and a large number of absentee ballots. Whatever the reason, iIt was days before anyone could project the winners in the state, and the results won't be officially certified until Nov. 20.
But while everyone else was wondering how Florida had managed to make itself a spectacle again, Larry Aultman, CIO of the Florida Department of State, and his team were witnessing success after 17 months of hard work.
"From my point of view, it was kind of boring," Aultman says. "We had the whole staff on hand, but we didn't have any failures. I was very pleased with the way the election ran from a technical standpoint. I don't make the votes, I just count them."
IT Breaks Down Silos
Fresh from a nearly 16-year stint as the president and CEO of financial services provider BizBanc, Aultman took on the mantle of CIO of the Florida Department of State in June 2011 with a broad mandate to overhaul IT. At the time, the department consisted of the Office of the Secretary and five divisions: the Division of Corporations, Division of Cultural Affairs, Division of Elections, Division of Historical Resources and Division of Library and Information Services. Each division had its own IT staff.
Aultman's task was to break down those silos and build a new division, the Division of Administrative Services, which would provide IT to the other five divisions.
"The first thing I did was to consolidate all five divisions into a single division so I could get my hands on what was going on and so I could figure out what kind of talent we had, what projects were in the work, that sort of thing," Aultman says, noting that he came in at the end of three years of compounded budget cuts and staff reductions. "We were really feeling the strain."
At the same time, he had 17 months to overhaul the Florida Voter Registration System, the electronic repository of all registered voters in the state. The system is also responsible for collecting all election data as it comes in from the counties. Aultman notes that Florida law requires the counties to provide the Department of State's Division of Elections with election data as it occurs, which is then presented to the public in real time through the Florida Election Watch website.
"In the past, the counties provided data when they wanted to and pretty much how they wanted to," Aultman says. "The legislature has put new controls in place that caused them to provide good data in real time."
The Election Watch website had been built by a contractor in 2009, but it experienced significant issues during the subsequent gubernatorial election in 2010.
"In June of 2011 when I came to the Department of State, Secretary Browning told me, 'I do not want the system to go down,'" Aultman says. "He did not want the system to be unavailable to the public. He wanted redundancy and failover and, of course, high availability. He wanted the system to have accountability within it and automation to the highest degree so we could work with the counties to get the data in in good form. He did not want the election system to provoke a call from the governor to find out what was going on."
Finding Cloud Success With Windows Azure
Seventeen months later, on election night, Aultman knew he was witnessing the success of his plan. He and his team had built a 100% cloud-based system on Windows Azure, and it was performing perfectly. At 8 p.m. (an hour after the polls had officially closed, though those who had been in line by 7 p.m. would continue to wait for hours before casting their ballots), the Election Watch website was sustaining 11,000 simultaneous connections, about 70% of which were new views every minute. It climbed to 23,000 simultaneous connections throughout the night and was still at 18,000 active connections at midnight. It was still at 5,000 the next morning.
"We didn't miss a beat," Aultman says. "Our error rate was 0.01%. The system had automatic retries, so all customers got their pages. It was a very good experience. We served the public very well from a technical standpoint. Also, all the counties were able to upload their files. We ran a separate system for the Supervisors of Elections to upload their files and we didn't have any errors at all during the evening."
Voter Registration System Unseats COBOL, Elects .NET
Achieving that success was no simple matter.
"That system was 20-years-old when I came in and was losing all vendor support," Aultman says of the system that supported the Division of Corporations, the most active corporate filing entity in the U.S., which maintains in excess of eight million records. "I looked at that and realized I was going to have to rewrite the Division of Corporations and rewrite the Voter Registration System in an election year and support all the other divisions as well."
And the problems didn't stop there. Just about everything in Aultman's purview had been written in COBOL, DB6 and antiquated languages and few if any people under him had modern skills.
"The first thing I did was to get $100,000 from the Secretary of State's office to put my staff into classes," Aultman says. "I brought in consultants to teach them to program in a modern coding language."
He had his staff of 40 spend eight hours a day for five days learning object-oriented programming, primarily in the Microsoft .NET framework. After the week-long crash course, he had them spend half of their day for another six months getting their skills up to snuff.
"There were a whole bunch of nonbelievers," he says. "I heard every excuse under the sun. And I lost 15 to 20% of the staff to taking their retirement or moving to another agency because they couldn't or didn't want to learn what we had to learn. I told them you either learn or we don't need you."
Department of State Finds Agility in the Cloud
In the meantime, Aultman had been thinking about what needed to be done. He knew that to succeed in the tasks that had been set before him, he needed to take the Department of State into the cloud.
"I didn't have any intent when I got there to take them to the cloud," he notes, even though he had been responsible for bringing BizBanc into the cloud.
"My intent was to fix what was there. But the money situation, the coding situation, the whole cost of trying to fix what was there and do it in a timely fashion just did not allow me much choice. Looking at the money and the time and the talent, it seemed like a reasonable way to go. I knew how to do a cloud project. I'd done it before. I knew what it could save if done correctly and I knew it could not save anything if done incorrectly," Aultman says.
The more he looked at it, the more he knew taking Florida's applications to the cloud was the best choice. Government, he says, particularly Florida's government, takes a monolithic approach to software. It wants to plan the software, construct it and then maintain it like a concrete building.
"Software just doesn't work that way anymore," he says. "You need agility. You need the ability to change with the times as technology changes. You need a way to provide applications that are less monolithic and more product-like. But that's counter to what the state does. It creates applications that run for a very long time doing things that seemingly don't change. But the law changes practically at every session. The judges rule on various things that force change on the system. The laws for corporations changes. This happens very, very frequently. We need a way that applications can be changed without so much cost, without so much testing, without so much human input."
Aultman decided Microsoft's Windows Azure was the platform he needed. His staff now had the .NET skills they needed, and the platform provided a great deal of ready-made code that he wouldn't have to replicate.
"I did hire some top guns to help me write the core set of services," he says. "The notion was that I would build a set of services on top of the Azure platform. It uses Azure, but it abstracts Azure so that if I needed to move it somewhere else, I could without a wholesale rewrite."
The system has provided a number of benefits, not the least of which is decoupling storage from compute, allowing IT to provision those resources at need rather than building for peak utilization.
Cloud Will Save Florida $1 Million a Year
The cloud-based Voter Registration System has not provided any savings to date, Aultman notes, though that's primarily because he's required to keep the old legacy systems up and running during the transition.
"Next year, I intend to retire the old database," he says. "Our licensing and support costs for that runs us about $1.5 million, not including personnel and contractors. I expect over the next three years, we'll save $1 million a year roughly in just licensing and operations costs."
Perhaps more importantly, the higher degree of automation allows him to run a leaner ship. He notes that he's already down to a headcount of 30 from the 40 people that he started with 17 months ago, and he expects he'll be forced to cut back his staff by another 10% to 15% in the next few years.
"That's significant," he says. "I don't know what next year will bring. Before I got here we lost all our network support and infrastructure support people due to data center consolidation. The consolidation of all our IT services was partly as a result of loss of staff. The whole change to application development and moving to the cloud is an attempt to provide the same level of service with fewer people."
For now, it's working, he says.
"They have really, really come so far in the last year," Aultman says of his staff. "You've got to get the combination right. You can't just go to the cloud. It doesn't make any sense by itself. You've got to do the other things. You've got to think about how you recognize projects, how you implement them, how you plan for their retirement. Our method is to pair people up with people who have different skillsets. We're working on real code reviews, not beating each other up, but more of a training thing. I'm making it something you want to do. You want to have a code review and learn from that. We're putting people in small teams of three, not six, and changing those teams frequently."
Over the next 12 months, Aultman's plan is to continue to modernize the Voter Registration System to make it even more resilient. Also, he wants to provide edge connections that will allow the data to be checked by the county software before it gets to the Division of Elections, kicking back errors for corrections in a much more timely fashion.
Thor Olavsrud covers IT Security, Big Data, Open Source, Microsoft Tools and Servers for CIO.com. Follow Thor on Twitter @ThorOlavsrud. Follow everything from CIO.com on Twitter @CIOonline and on Facebook. Email Thor at tolavsrud@cio.com
Read more about government in CIO's Government Drilldown.