December 01, 2009, 7:49 PM — If you're working in a hurricane hot zone, chances are you've become accustomed to thinking on your feet. That sensibility served Bryan Sastokas, CIO of the city of Coral Springs, Fla., well as his IT department traversed the murky waters of implementing a virtual infrastructure supporting critical public safety and other applications. In this interview with contributing writer Beth Schultz, Sastokas reflects on ...
MIGRATION STRATEGY: When we started out, we thought we'd do our application migration from the current physical servers to virtual servers. That was going OK, but it wasn't very smooth. It was somewhat kludgy. About a month into the deployment, we decided building a fresh environment, and then porting the applications, would be better.
ROOTING OUT PROBLEMS: When we did the migration from the physical box to a virtual environment, application performance had been unacceptable, but we weren't sure why. We thought maybe we were building the environment incorrectly or maybe that we were missing something. We thought, too, the uniqueness of the applications -- city applications aren't always your typical shrink-wrapped packages; some are custom-made for specific public safety tasks -- could have been a problem. So we pulled the plug on our original deployment plans, went with clean builds, and that was 100% successful for us. We never did figure out the root cause, though.
LAB VS. PRODUCTION: We did test the applications prior to the physical-to-virtual migration and, in the test environment, they worked fine. We saw no performance degradation, had no servers crash. Things were very stable. But in the production environment, where migration happened, that wasn't the case. That's why we stepped back and decided the problems had to do with the way we were implementing the virtual environment.
PUTTING VENDORS TO THE FIRE: In the past, we never would pay much mind to the infrastructure. We'd just know the application would be running on a quad-core processor, with 16 gigabytes of RAM, say, and it was simple. But with the virtual environment, without the one-to-one relationship, we had to make it explicit to our vendors, 'Here's the service level I'm providing to my residents -- my clients -- with this availability. Your product is going to operate in this environment, and you have to commit to us that you're going to be able to keep these services available or, if they fail, that you can restore quickly.
LICENSING: What we didn't expect was the problem we had with some of our specialized applications that had different licensing structures. That did catch us off guard. Some of them had licensing keys that were written by serial port and you had to have them plugged into the physical device, but now we have virtual servers. How do you relate that serial key back to the server? We worked this all out by building that vendor relationship, but ahead of time we should have had a better idea on the licensing.
RUNAWAY MANAGEMENT: I would advise anybody looking to get into virtualization to invest in a management tool for figuring out capacities available on the virtual hosts. We use a tool that helps us figure out, in the development lab, which physical box has the capacity to house an application of a particular nature. We knew that once we straightened out the deployment piece that the possibility of runaway management would be a real challenge because spinning up new virtual machines is so easy to do.
PHYSICAL-TO-VIRTUAL RATIO: It really does depend on the application, but we've learned that we can get up to almost 30 applications on a physical box without any problem. These are basic, simple applications. But for databases, we try to lower that to no more than 10. We try
to get the physical box to as close to 100% utilization as possible without impacting queries -- that's the beauty of virtualization, it is letting us squeeze every ounce of CPU utilization, every byte of memory and terabyte of storage out of these boxes.
What do you know now that you wish you'd known then? Share your tales here or contact Beth Schultz, at bschultz5824@gmail.com.
