Virtualization is moving further into the datacenter all the time, with even critical large-scale applications like Exchange and databases now being virtualized. Whether you're using virtualization to make large applications more manageable or to consolidate many small applications, a server with lots of RAM, lots of processor cores, and lots of I/O is a good thing. And so is a SAN packed with features that ease the management of storage for virtual machines.
From a storage viewpoint, each virtual machine uses a file to simulate a physical hard disk. This file, a VMDK (Virtual Machine Disk Format) under VMware or VHD (Virtual Hard Disk) under Microsoft, can be located on a server's internal drive or on a SAN. There are several advantages to putting the file on a SAN: The file can be duplicated using the storage's snapshot function, the file can be moved easily from one hardware server to another for scalability or fault tolerance, and the storage itself can be more easily made fault tolerant.
[ See the Test Center reviews: VMware vSphere 4: The once and future virtualization king | Microsoft's Hyper-V R2 is hot on VMware's heels. ]
I tested five midrange SAN systems that deliver the goods for virtual environments: the Compellent Storage Center 4.0, the Dell EqualLogic PS4000, the HP StorageWorks 2000sa G2 Modular Smart Array, the Pillar Axiom 600 from Pillar Data Systems, and a build-it-yourself pairing of the Promise vTrak E610f hardware and DataCore's SANmelody 3.0 storage software. This review focuses on these systems' virtualization-friendly features. For a close look at the full range of their capabilities, see Network World's companion review.
The prices (as tested) of these systems range widely, from less than $10,000 for the Promise Technologies and DataCore combination to $130,000 for the Pillar Data system. All the manufacturers have models ranging from inexpensive starters to very high-performance datacenter-ready systems. As you can tell from the range in price, the models I tested don't necessarily compete with each other.
Virtual Machine magic Using a snapshot to duplicate a VMDK or VHD file bypasses the difficulty of backing up a virtual disk while the VM is running. It also makes it easy to clone a working VM, convert the snapshot to a new volume, and mount the new volume as a new VM. The process of converting snapshots to a new volume and mounting the new volume varies widely among the SANs tested, not so much in the procedure involved, but in the time required to unify all the snapshots into a new version of the volume. In testing, I created a 100GB volume, put several VMDK files on it, made changes to the VMs over several days, then cloned the volume and mounted the clone as a new volume. Depending on the storage system, the time required to create the new volume ranged from a few seconds to almost 20 minutes.
My five SANs also differed in their connectivity options, a factor that made an apples-to-apples comparison of I/O performance infeasible. For example, the HP is Fibre Channel only, the Dell is Ethernet (iSCSI) only, and the other three systems support both Fibre Channel and Ethernet. My testing therefore focused on features and functionality rather than IOps, which can vary widely even across SAN models from the same manufacturer.
Fibre Channel versus iSCSI is one of the major decisions you'll need to make when configuring a SAN. In the virtualization space, Fibre Channel has an edge if you want to put a lot of VMs on a single server. This is because network port count and throughput tend to be bottlenecks anyway; if you're using Ethernet ports for both storage and network, it's that much harder to get a lot of VMs running on the same server. With multiple VMs contending for a single iSCSI connection, performance can really suffer.
The way around this is to use multiple gigabit connections or one 10-gigabit Ethernet connection, but 10Gbps connections and switch ports aren't any cheaper than Fibre Channel, and you can stuff only so many four-port Ethernet cards into a server. The exception is when performance isn't as much a consideration as cost -- iSCSI can be less expensive.
If you install iSCSI and then need to increase performance, your options are limited. TOE cards -- TCP/IP offload Ethernet NICs engineered to provide high-performance iSCSI connections -- can cost more than 2Gbps or even 4Gbps Fibre Channel HBAs. The other option to increase iSCSI performance is to bond multiple gigabit ports, which again uses up ports on the server, the switch, and the storage system.
In addition to creating snapshots on a regular basis to capture changes to VMDK or VHD files, you may want to enable replication, either to another local storage system or to a remote system at another location. Doing this through the storage system rather than via add-on software on the hypervisor can save you quite a bit of money. These features come with many SAN systems (including all of those reviewed here except the HP, which provides snapshots but not replication) and can provide redundancy for multiple virtualization servers with no per-server cost.
Roll your ownIronically, the least expensive system in my test, the Promise/DataCore combination, is also the only one that includes a plug-in for VMware's management clients that allows you to provision, share, clone, replicate, and expand virtual disks among physical servers and VMs. DataCore will also let you try the software for free, as a virtual appliance that can run on Citrix XenServer or Microsoft Hyper-V, working with up to a terabyte of storage.
On the other hand, although DataCore SANmelody offers an extra degree of integration with most virtualization platforms, it may make many administrators nervous. Because SANmelody runs on top of Windows and uses whatever storage the server can see, assembling a high-performance system will require careful planning and research into the most appropriate hardware. If there are problems, there's no single support number; the Promise storage system, the separately purchased hard drives, and the Windows and DataCore software are all supported separately.
Each of the systems I tested has its place, and each will suit some administrators. The Promise/DataCore system is very inexpensive, while nevertheless providing high-end storage features and great integration with most virtualization platforms. The HP offers great performance and good basic storage features. The Dell/EqualLogic systems offer lots of bang for the buck in storage features, although iSCSI and virtualization aren't always the best companions. The Compellent system offers a very mature and stable platform, with very high performance and sophisticated but easy-to-use features. Pillar Data offers a unique application-aware storage system with great performance, albeit at a significantly higher price than the other systems tested.
If you have an environment where you need to be able to run a bunch of different operating systems and don't require high performance, but want to be able to clone images from a gold set and boot the clones, the Promise/DataCore combo or the Dell/EqualLogic pair are great choices. Both provide a lot of capacity with all the features you'd want for a low cost. If you're looking for higher performance with low cost and don't care about replication for disaster recovery, the HP is a good fit. If you have a number of storage admins or virtualization admins that you want to be to able to provision storage without a lot of training, the Pillar Data system is an excellent though pricey choice. Finally, the all-around winner is the Compellent system, with a combination of flexibility, sophisticated features, high performance, and superior ease of use.
Compellent Storage Center 4.0 Compellent has done very well in previous tests, twice winning an InfoWorld Technology of the Year Award. It's once again in contention with an excellent overall system that combines very high performance, good capacity, great ease-of-use, and an excellent feature set.
The system I received for testing included a storage controller and two shelves of drives, a tier-one high-performance set with sixteen 300GB 15,000-rpm drives, and a tier-three high-capacity set with eight 500GB SATA drives (and room for eight more), for a total usable capacity of just over 8.5TB. All three boxes included dual controllers and dual power supplies for a fully redundant, dual-path 4Gbps Fibre Channel system. The system included all software options: server instant replay (snapshots), dynamic capacity (thin provisioning), data progression (automatic migration of data to lower tiers), and synchronous and asynchronous replication. The as-tested price of $68,709 is the second highest in this test, but not exorbitant for what you get.
Creating volumes through the browser-based admin console is simple and straightforward, and the virtualization-specific task of coalescing a number of snapshots into a new volume took less than 10 seconds for a 100GB volume. Setting up replication, even to a remote system at another location, is also a snap. The automatic migration of data to lower tiers ensures that the highest-performance drives do the heavy lifting, while data that isn't used regularly is migrated to less-expensive SATA-based storage. The system offered eight 4Gbps Fibre Channel connections, enough to support multiple VMware servers with sixteen or more VMs per server without storage bottlenecks.
Compellent also includes a full PowerShell plug-in command set for both Hyper-V and VMware environments. PowerShell, the scripting tool included with Windows Server 2008, allows for easy automation of tasks from Windows systems. The plug-in command set allows an admin to script tasks such as provisioning LUNs and volumes, configuring LUN masking, formatting volumes, installing VMs through Hyper-V Manager or VMware's Virtual Center (or vCenter Server), and installing images -- pretty much all the tasks you might want to automate when provisioning storage along with a virtual environment. The system supports full interaction with PowerShell to pull information on available volumes and perform error checking. This requires some basic programming, but still allows for complete provisioning of both Hyper-V and VMware VMs, as well as their storage, through one interface.
Compellent has another unique feature: the white-space recovery tool. Windows' NTFS is constructed to write new files to any unused blocks on disk before writing over blocks that have been freed up by deleting files. This doesn't matter on a normal hard disk, but will cause thin-provisioned virtual volumes to quickly expand to their maximum capacity. Compellent's recovery tool searches for blocks that are part of deleted files and marks them as unused so that Windows will overwrite them rather than using up all available space, thus ensuring that volumes don't expand unnecessarily.
Compellent Storage Center offers a great combination of capacity, high performance, and sophisticated features, making it hard to beat, whether for a traditional SAN environment or when specifically supporting virtualization. It's not the cheapest, but if you want to run a lot of VMs with both good performance and fault tolerance, it's an excellent choice.
Dell EqualLogic PS4000 Dell has a wide variety of storage systems, from simple direct-attached SCSI boxes to the EqualLogic iSCSI systems and Fibre Channel storage. The EqualLogic PS4000 is a very usable iSCSI system that starts at around $10,000, with 16 250GB SATA drives. I received two systems, both with dual controllers and dual power supplies, at a cost of $17,000 each as tested. Each controller has two gigabit Ethernet ports for iSCSI and a management port. The systems are available with up to 16 1TB drives.
After installing the first PS4000, adding the second one was simply a matter of turning it on. The admin console found the second system, added it to the pool of available storage, and was ready to load balance across both units very quickly.
The PS4000 offers a great feature set included at the base price, with the usual thin provisioning, snapshots, replication, and load balancing across multiple iSCSI ports and multiple arrays to optimize performance. Creating new volumes and setting up options like replication is very straightforward. Coalescing multiple snapshots into a new volume was quick and easy. Running more than six VMs under load on a single volume on one PS4000 system resulted in disk access bottlenecks. Running four iSCSI connections across two systems fixed the problem, but used four out of eight gigabit Ethernet ports on the VMware server just for storage, leaving only four ports for network traffic.
The PS4000 offers an inexpensive starting price, a nice feature set, and the ability to grow in both capacity and performance by adding relatively inexpensive additional systems. While you won't be able to host as many VMs on a single server as Fibre Channel will allow, the starting price is lower because no Fibre Channel HBAs or switches are required.
HP StorageWorks 2000fc G2 Modular Smart Array The HP is the first storage system I've tested that uses 2.5-inch drives. With the small drives, HP can squeeze 24 disks into a 2U chassis rather than the usual 16 disks into a 3U chassis. The drives included with the test system were 73GB 15,000-rpm SAS drives, for a total capacity of just over 1.5TB formatted, which was priced at $13,000. The system included two 4Gbps Fibre Channel RAID controllers, dual power supplies, and a good basic feature set. It doesn't offer all the features of the more expensive systems, but at $13,000 for a blazing fast system, it offers great performance for the dollars.
Setup of the system was straightforward, with a simple Web-based interface and the features a virtualization administrator would most want to see, particularly snapshots and the ability to clone volumes to create additional VMs. Coalescing multiple snapshots into a single new volume went very quickly, and the overall speed of the system was able to support 16 VMs on a VMware server without bottlenecks.
Pillar Data Systems Axiom 600 The Pillar Data AX600 is not unusual physically, with three components, a management unit (called a Pilot), a controller head (the Slammer), and drive shelves (Bricks), but Pillar Data's approach to storage provisioning and management is definitely unusual. The company calls the approach "application-aware storage," which means that rather than specifying a volume that is RAID5 with a 64KB stripe and scheduled snapshots, you specify an application type -- Oracle Database or SQL Server or VMware ESX hypervisor, for example -- and the system sets up the volume to optimize the performance for that app. A variety of application profiles are included with the system, and more specialized ones are available from the Pillar Data support site.
Rather than specifying a storage tier (SSD, 15K, 10K, SATA), you specify a performance class (premium, high, medium, low, or archive), and the system picks the best storage on which to place the volume, based on what's available. These storage options not only include the speed of the disks the volume is on, but also the place on disk (outer, middle, or inner tracks) and CPU utilization, amount of cache, and allocated bandwidth. The system even includes a predictive analysis module that can analyze the impact on existing storage volumes of a new volume, depending on the performance requirements and application type it is given.
The Pillar Data system was the most expensive of the ones I tested, but it also had the highest capacity, multiple paths throughout for high availability, a very complete feature set, and great usability. Coalescing multiple snapshots into a new image went very quickly, as did mounting the new volume. The system would easily support multiple VMware servers without any storage bottlenecks.
Promise vTrak E610f and DataCore SANMelody 3.0 The build-it-yourself Promise and DataCore system represents how far you can go on a tight budget if you're willing to roll up your sleeves. The Promise vTrak E610f is a 16-drive enclosure that includes two 4Gbps Fibre Channel controllers that offer RAID levels 0, 1, 0+1 (or 10), 5, 6 (double parity drive), 50 (two mirrored RAID 5 volumes), or 60. You can add up to three additional enclosures via Serial Attached SCSI, all using the same dual controllers or as many Fibre Channel dual-controller systems as your switch will support. The Promise system doesn't include drives -- you add your own SAS or SATA drives. Installing drives is straightforward and shouldn't bother anyone with hardware experience; there's even a full paper manual that fully documents the process.
Once the drives are installed, connecting to the system and initializing the RAID set or sets is straightforward. I installed eight Western Digital Raid Edition 500GB SATA drives and eight Seagate Savvio 15,000-rpm 73GB SAS drives, creating a 280GB RAID10 volume with the Savvio drives and a 3.5TB RAID5 volume with the Western Digital drives.
The next step was installing the DataCore SANmelody software on an HP ProLiant ML370 G5 server. I didn't count the server in the cost of the system for two reasons. First, the server is still available for other uses, and second, a really expensive high-powered server is not required; SANmelody can even run on a VM. The server I used had a two-port QLogic 2Gbps HBA connecting it to the Promise array, but SANmelody will work with any storage the server can see -- whether direct-attached SCSI, internal storage, or external -- and can then make that storage available to other systems via Fibre Channel, iSCSI, or even Fibre Channel over Ethernet (FCoE).
Further, SANmelody provides the full set of storage management capabilities found in any of the other systems (apart from Pillar Data's application-aware automation), such as thin provisioning of volumes, snapshots, and synchronous and asynchronous replication. In addition, I configured automatic migration of data from the RAID10 high-performance volume to the high-capacity RAID5 SATA volume, essentially using the 15,000-rpm high-speed drives as cache for the SATA volume.
SANmelody also offers a plug-in for the VMware Infrastructure and vSphere clients. This lets you perform any storage-related task, including creating volumes, cloning existing volumes, and expanding disks from within the VMware management console. You can also download a free evaluation version of the software that will support up to a terabyte of storage.
With a total price of less than $10,000 for a full-featured storage system with 3.5TB of usable disk space and very high performance due to the 15,000-rpm drives, why would anyone pay more for a storage system? Peace of mind and simplicity of installation are the primary reasons. You have to build this system yourself, and if something goes wrong, there is no single throat to choke; you could have several different vendors pointing fingers at each other. One other issue is the speed of the SANmelody management system: The process of coalescing multiple snapshots into a single new volume, which took from a few seconds to about a minute with the other systems I tested, took nearly 20 minutes for this system to complete.
I expect that SANmelody's volume unification depends on the speed of the Windows server hosting it. The system I installed SANmelody on had one single-core 2.8GHz CPU and 2GB of RAM. By comparison, the Compellent controller has dual Intel Nehalem processors and 32GB of RAM, and is probably handling some operations with dedicated hardware.
If you're willing to invest the time in making sure the components are all correct, and in getting all the separate parts working together, the Promise-DataCore system offers a huge bang for the buck and great virtualization support to boot.
Recent articles on virtualization:
- Virtually there: The state of datacenter virtualization
- How to reuse your old virtual hosts
- Why Microsoft is sabotaging desktop virtualization
- The perfect storm of bad news for VDI
- Has VMware lost its mojo?
- The virtual virtualization case study
InfoWorld Test Center reviews:
- Microsoft's Hyper-V R2 is hot on VMware's heels
- VMware vSphere 4: The once and future virtualization king
- VMware View is good news, bad news
- Citrix hits the VDI high notes
- Microsoft's Hyper-V does the trick
- VMware pumps up VI3
This story, "Test Center: SANs tuned for virtualization pack nice surprises," was originally published at InfoWorld.com. Follow the latest developments in virtualization at InfoWorld.com.
This story, "Test Center: SANs tuned for virtualization pack nice surprises" was originally published by InfoWorld.
A compilation of one Linux expert's adventures with 10 desktop environments.
Which kind are you? Choose wisely!
It's taken a long time and the promise of SteamOS, but more and more big-name games are finally...
Trust and IT go hand in hand. Here are the red flags to watch for before you get burned.
Intel is looking to partners to help make its mark in smartphones
The Mobile Accessibility Checker flags potential accessibility issues with mobile apps
InfiniFlash is Sandisk's new all-flash platform for applications such as video streaming or...