All phone calls become electronic records, storing in log information such as how long the call took, when it was made, how the caller used the system and if anything unusual occurred during the call. The company's Microsoft SQL Server database holds all log information and keeps billing records.
Vail processes more than 48 million billing records a day, and while each call log is relatively small, they add up to about 4.5GB of data in the SQL database per day.
Customer automated queries had required significant disk access at Vail because the server's memory could not handle random access across the whole database. The databases are mirrored for redundancy, but while redundant writes occurred, they effectively blocked read operations.
Prior to turning to flash storage, Vail had been using standard Dell servers with 12 10,000 rpm SAS drives set up in a RAID 10 configuration. The spindles didn't provide the response times the company needed. For example, response time for customers calling to check billing-records averaged five seconds, Fruin said.
"With SQL, if you're doing a query and you're looking for something that's not indexed, the SQL engine has to go to disk and inspect it and see what query you're looking for. For us, even one day's worth of data could be 100,000 records," Fruin said. "With no index, there are many seeks, and seeks are what kills performance in database queries."
Fruin said the company had to eliminate the disk-drive bottleneck without changing the architecture of the billing system or the customer-care interface. "Re-engineering the billing system on a No-SQL solution wouldn't achieve these goals," he said.
Thus, Fruin selected flash storage, first by adding 2.5-in Intel SSDs to the Dell servers, and then PCIe modules from Virident Systems. The SSDs improved performance, but only about four times. The two 1TB Virident flash modules offered up a 10x performance improvement over hard drives.
Virident's FlashMAX PCIe modules come as either high cost, high performance single-level cell (SLC) NAND flash or lower-performance, lower cost multi-level cell (MLC) NAND flash devices. They range in capacity from 550 gigabytes to 2.2 terabytes per module. The MLC cards, which Fruin chose, offer read performance of up to 1.3GBs or 325,000 IOPS using 4k blocks.
Fruin said the flash allowed 10 times as many caller queries to be handled in the same amount of time.
"We were looking for four times improvement based on projected call volume increase over time and we got 10X, so we were surprised," he said.
After Virident's FlashMAX flash module was installed, response times to billing inquiries dropped from an average of five seconds to virtually real time -- as low as 500 milliseconds.