Virtualization is moving further into the datacenter all the time, with even critical large-scale applications like Exchange and databases now being virtualized. Whether you're using virtualization to make large applications more manageable or to consolidate many small applications, a server with lots of RAM, lots of processor cores, and lots of I/O is a good thing. And so is a SAN packed with features that ease the management of storage for virtual machines.
From a storage viewpoint, each virtual machine uses a file to simulate a physical hard disk. This file, a VMDK (Virtual Machine Disk Format) under VMware or VHD (Virtual Hard Disk) under Microsoft, can be located on a server's internal drive or on a SAN. There are several advantages to putting the file on a SAN: The file can be duplicated using the storage's snapshot function, the file can be moved easily from one hardware server to another for scalability or fault tolerance, and the storage itself can be more easily made fault tolerant.
[ See the Test Center reviews: VMware vSphere 4: The once and future virtualization king | Microsoft's Hyper-V R2 is hot on VMware's heels. ]
I tested five midrange SAN systems that deliver the goods for virtual environments: the Compellent Storage Center 4.0, the Dell EqualLogic PS4000, the HP StorageWorks 2000sa G2 Modular Smart Array, the Pillar Axiom 600 from Pillar Data Systems, and a build-it-yourself pairing of the Promise vTrak E610f hardware and DataCore's SANmelody 3.0 storage software. This review focuses on these systems' virtualization-friendly features. For a close look at the full range of their capabilities, see Network World's companion review.
The prices (as tested) of these systems range widely, from less than $10,000 for the Promise Technologies and DataCore combination to $130,000 for the Pillar Data system. All the manufacturers have models ranging from inexpensive starters to very high-performance datacenter-ready systems. As you can tell from the range in price, the models I tested don't necessarily compete with each other.
Virtual Machine magic Using a snapshot to duplicate a VMDK or VHD file bypasses the difficulty of backing up a virtual disk while the VM is running. It also makes it easy to clone a working VM, convert the snapshot to a new volume, and mount the new volume as a new VM. The process of converting snapshots to a new volume and mounting the new volume varies widely among the SANs tested, not so much in the procedure involved, but in the time required to unify all the snapshots into a new version of the volume. In testing, I created a 100GB volume, put several VMDK files on it, made changes to the VMs over several days, then cloned the volume and mounted the clone as a new volume. Depending on the storage system, the time required to create the new volume ranged from a few seconds to almost 20 minutes.
My five SANs also differed in their connectivity options, a factor that made an apples-to-apples comparison of I/O performance infeasible. For example, the HP is Fibre Channel only, the Dell is Ethernet (iSCSI) only, and the other three systems support both Fibre Channel and Ethernet. My testing therefore focused on features and functionality rather than IOps, which can vary widely even across SAN models from the same manufacturer.
Fibre Channel versus iSCSI is one of the major decisions you'll need to make when configuring a SAN. In the virtualization space, Fibre Channel has an edge if you want to put a lot of VMs on a single server. This is because network port count and throughput tend to be bottlenecks anyway; if you're using Ethernet ports for both storage and network, it's that much harder to get a lot of VMs running on the same server. With multiple VMs contending for a single iSCSI connection, performance can really suffer.
The way around this is to use multiple gigabit connections or one 10-gigabit Ethernet connection, but 10Gbps connections and switch ports aren't any cheaper than Fibre Channel, and you can stuff only so many four-port Ethernet cards into a server. The exception is when performance isn't as much a consideration as cost -- iSCSI can be less expensive.
If you install iSCSI and then need to increase performance, your options are limited. TOE cards -- TCP/IP offload Ethernet NICs engineered to provide high-performance iSCSI connections -- can cost more than 2Gbps or even 4Gbps Fibre Channel HBAs. The other option to increase iSCSI performance is to bond multiple gigabit ports, which again uses up ports on the server, the switch, and the storage system.
In addition to creating snapshots on a regular basis to capture changes to VMDK or VHD files, you may want to enable replication, either to another local storage system or to a remote system at another location. Doing this through the storage system rather than via add-on software on the hypervisor can save you quite a bit of money. These features come with many SAN systems (including all of those reviewed here except the HP, which provides snapshots but not replication) and can provide redundancy for multiple virtualization servers with no per-server cost.
Roll your ownIronically, the least expensive system in my test, the Promise/DataCore combination, is also the only one that includes a plug-in for VMware's management clients that allows you to provision, share, clone, replicate, and expand virtual disks among physical servers and VMs. DataCore will also let you try the software for free, as a virtual appliance that can run on Citrix XenServer or Microsoft Hyper-V, working with up to a terabyte of storage.
On the other hand, although DataCore SANmelody offers an extra degree of integration with most virtualization platforms, it may make many administrators nervous. Because SANmelody runs on top of Windows and uses whatever storage the server can see, assembling a high-performance system will require careful planning and research into the most appropriate hardware. If there are problems, there's no single support number; the Promise storage system, the separately purchased hard drives, and the Windows and DataCore software are all supported separately.
Each of the systems I tested has its place, and each will suit some administrators. The Promise/DataCore system is very inexpensive, while nevertheless providing high-end storage features and great integration with most virtualization platforms. The HP offers great performance and good basic storage features. The Dell/EqualLogic systems offer lots of bang for the buck in storage features, although iSCSI and virtualization aren't always the best companions. The Compellent system offers a very mature and stable platform, with very high performance and sophisticated but easy-to-use features. Pillar Data offers a unique application-aware storage system with great performance, albeit at a significantly higher price than the other systems tested.
If you have an environment where you need to be able to run a bunch of different operating systems and don't require high performance, but want to be able to clone images from a gold set and boot the clones, the Promise/DataCore combo or the Dell/EqualLogic pair are great choices. Both provide a lot of capacity with all the features you'd want for a low cost. If you're looking for higher performance with low cost and don't care about replication for disaster recovery, the HP is a good fit. If you have a number of storage admins or virtualization admins that you want to be to able to provision storage without a lot of training, the Pillar Data system is an excellent though pricey choice. Finally, the all-around winner is the Compellent system, with a combination of flexibility, sophisticated features, high performance, and superior ease of use.
Compellent Storage Center 4.0 Compellent has done very well in previous tests, twice winning an InfoWorld Technology of the Year Award. It's once again in contention with an excellent overall system that combines very high performance, good capacity, great ease-of-use, and an excellent feature set.
The system I received for testing included a storage controller and two shelves of drives, a tier-one high-performance set with sixteen 300GB 15,000-rpm drives, and a tier-three high-capacity set with eight 500GB SATA drives (and room for eight more), for a total usable capacity of just over 8.5TB. All three boxes included dual controllers and dual power supplies for a fully redundant, dual-path 4Gbps Fibre Channel system. The system included all software options: server instant replay (snapshots), dynamic capacity (thin provisioning), data progression (automatic migration of data to lower tiers), and synchronous and asynchronous replication. The as-tested price of $68,709 is the second highest in this test, but not exorbitant for what you get.
Creating volumes through the browser-based admin console is simple and straightforward, and the virtualization-specific task of coalescing a number of snapshots into a new volume took less than 10 seconds for a 100GB volume. Setting up replication, even to a remote system at another location, is also a snap. The automatic migration of data to lower tiers ensures that the highest-performance drives do the heavy lifting, while data that isn't used regularly is migrated to less-expensive SATA-based storage. The system offered eight 4Gbps Fibre Channel connections, enough to support multiple VMware servers with sixteen or more VMs per server without storage bottlenecks.
Compellent also includes a full PowerShell plug-in command set for both Hyper-V and VMware environments. PowerShell, the scripting tool included with Windows Server 2008, allows for easy automation of tasks from Windows systems. The plug-in command set allows an admin to script tasks such as provisioning LUNs and volumes, configuring LUN masking, formatting volumes, installing VMs through Hyper-V Manager or VMware's Virtual Center (or vCenter Server), and installing images -- pretty much all the tasks you might want to automate when provisioning storage along with a virtual environment. The system supports full interaction with PowerShell to pull information on available volumes and perform error checking. This requires some basic programming, but still allows for complete provisioning of both Hyper-V and VMware VMs, as well as their storage, through one interface.
Compellent has another unique feature: the white-space recovery tool. Windows' NTFS is constructed to write new files to any unused blocks on disk before writing over blocks that have been freed up by deleting files. This doesn't matter on a normal hard disk, but will cause thin-provisioned virtual volumes to quickly expand to their maximum capacity. Compellent's recovery tool searches for blocks that are part of deleted files and marks them as unused so that Windows will overwrite them rather than using up all available space, thus ensuring that volumes don't expand unnecessarily.
Compellent Storage Center offers a great combination of capacity, high performance, and sophisticated features, making it hard to beat, whether for a traditional SAN environment or when specifically supporting virtualization. It's not the cheapest, but if you want to run a lot of VMs with both good performance and fault tolerance, it's an excellent choice.
Dell EqualLogic PS4000 Dell has a wide variety of storage systems, from simple direct-attached SCSI boxes to the EqualLogic iSCSI systems and Fibre Channel storage. The EqualLogic PS4000 is a very usable iSCSI system that starts at around $10,000, with 16 250GB SATA drives. I received two systems, both with dual controllers and dual power supplies, at a cost of $17,000 each as tested. Each controller has two gigabit Ethernet ports for iSCSI and a management port. The systems are available with up to 16 1TB drives.
After installing the first PS4000, adding the second one was simply a matter of turning it on. The admin console found the second system, added it to the pool of available storage, and was ready to load balance across both units very quickly.
The PS4000 offers a great feature set included at the base price, with the usual thin provisioning, snapshots, replication, and load balancing across multiple iSCSI ports and multiple arrays to optimize performance. Creating new volumes and setting up options like replication is very straightforward. Coalescing multiple snapshots into a new volume was quick and easy. Running more than six VMs under load on a single volume on one PS4000 system resulted in disk access bottlenecks. Running four iSCSI connections across two systems fixed the problem, but used four out of eight gigabit Ethernet ports on the VMware server just for storage, leaving only four ports for network traffic.
The PS4000 offers an inexpensive starting price, a nice feature set, and the ability to grow in both capacity and performance by adding relatively inexpensive additional systems. While you won't be able to host as many VMs on a single server as Fibre Channel will allow, the starting price is lower because no Fibre Channel HBAs or switches are required.