In July, the electronic payments company posted a job listing on the Data Center Job Board for a senior facilities engineer in Virginia to ensure the smooth operation and launch of new state-of-the-art data centers.
Visa is looking to mimic the 2006 launch of its Operations Center Central (OCC) processing facility in Denver in other data centers around the globe. Visa is pushing the envelope of virtualization in that facility and two others in different locations.
"OCC is an ongoing project," says Andy Lewis, Visa's head of global engineering at the Denver OCC. "Just as we complete one aspect of it there's going to be something else that comes along that is a further enabler to reduce costs and increase reliability and availability, while managing risk at the same time. Is there ever an end state?"
Its business would say no. Visa's 1.6 billion global card-holders account for more than US$3 trillion in annual transaction volume, which is growing at a 20% per-year clip. VisaNet, the company's global network, serves as the backbone for roughly one-seventh of American consumer expenditure.
Each OCC data center will handle more than $1 trillion in annual transaction volume and according to Visa was designed to meet the growing volume of electronic credit, debit and prepaid transactions for the foreseeable future. Network, server and storage virtualization are key to enabling that.
The OCCs run a single synchronized image of transaction processing around the globe. And each OCC runs multiple instances -- or virtual images -- of that single image within the data center. This gives Visa "redundancy within redundancy" for credit or debit authorization, and the ability to failover to another data center as well as internally within a data center: Visa engineers at an OCC can manage another data center thousands of miles away connected over the company's VisaNet global payments network.
Virtualization has also helped Visa manage costs. By allocating or replicating processing cycles logically, Visa's IT budget went up only 3% or 4%, and unit costs were cut by about 50% between 2000 and 2007, as transactions grew at that 20% annual clip, CIO Michael Dreyer said at Cisco conference last summer.
But Visa has taken a methodical approach to virtualization, and has adopted different, discrete approaches from all of its top vendors, starting with the IBM Multiple Virtual Storage (MVS) and Virtual Machine (VM) mainframes it's been using since the mid-1970s. The company also uses virtualization offerings from HP/Tandem, VMware and other server, storage and network vendors, Lewis says, to replicate, partition and allocate resources without purchasing and deploying additional physical hardware and software assets.
But the environment has to be right for it, Lewis admits. "Have we gone full bore with virtualization in our production, core systems?" Lewis asks. "No we have not.
"Virtualization is one of the tools in a toolbox we have that helps us lower costs, increase utilization, define flexibility among resources," he says. "Consumption-based pricing, and making sure we have the correct terms and conditions in place with our vendors for software and hardware all play a part in this increased utilization and better value per unit cost. How are you managing assets and utilization of assets? We look at how you can virtualize to gain more effective usage and consumption of those assets."
But there are many inhibitors to reaching an "end state" of virtualization in and between data centers, Lewis says, where every resource can be physically decoupled from its host machine. Among them:
-- Lack of standards -- Lewis says this is the biggest inhibitor as there are a lot of "niche, proprietary solutions" available.
-- Immaturity -- Many of the legacy hypervisor products are lower-level VM environments, integrated at the hardware and firmware level, Lewis says. The newer, higher-level software and application virtualization hypervisors are "more appealing" but "more risk," he says.
-- Charge-back mechanisms -- Managing the normal unit of work and charging back company departments for it is more challenging in a virtualized environment, Lewis says.
-- Shared environments -- The key to virtualization is how you share an environment with another department within the company. Organizationally, that could be an inhibitor, Lewis notes, because offering a single synchronized image breaks down barriers between constituencies.
-- Lack of an "aggregated protocol" -- Something akin to MPLS and its tunneling mechanisms is needed in the data center to converge ESCON, FICON, Fibre Channel, SCSI and Ethernet into a more operationally efficient fabric, Lewis says.
Specific to the standards issue, Visa has to define its own internal standards for managing and securing its virtual environment. Vendors are now offering only point products.
"We're making headway with VMotion from VMware, it's improved our ability to have awareness and vision into the usage and consumption and configuration of our environments," Lewis says. "But again, If I go down each vendor's path whether storage or server or network, each vendor has a different approach. There's some concern over who's strategy is actually going to win through, who's going to be the manager of managers and who's going to share their IP to ensure it's being managed effectively at the global and client level."
As for an aggregated protocol, vendors are working on standards for unified data-center fabrics, such as the Fibre Channel over Ethernet specification from Technical Committee T11 of the InterNational Committee for Information Technology Standards, and the Converged Enhanced Ethernet/Data Center Ethernet efforts from the IEEE, IBM, Cisco, Intel, EMC and others. But again, these are in prestandard form, and Visa is reluctant to adopt anything that has not been standardized.
"I'm going to be leery of it for a couple of years until I really think it's baked in as a standard," he says. "We saw this at a higher level [recently] with regard to network file access [and] I/O: Was it going to be SCSI over Ethernet? SCSI over IP? FCoIP? We will see continued progress in specific areas around ESCON-to-FICON, around Fibre Channel arbitrary looped and switched/switched2 fabrics. . . . But I can't see that there's any one silver bullet right now."
Visa is also evaluating newer virtualization products from Cisco, including the VFrame Data Center resource orchestration appliance, and Nexus 7000 switches. VFrame is an appliance designed to provision compute, network and storage resources together as virtual services through a policy engine that automates resource changes in response to infrastructure outages and performance alterations. Nexus 7000 optimized for high-density 10 Gigabit Ethernet in the data center, and supports a unified switching fabric designed to provide all servers with access to all network and storage resources.
But pricing and capacity issues and a mainframe legacy keep the company from implementing them for now, Lewis says.
"We do use mainframe, nonstandard systems" for data center orchestration, he says. "Tandem/HP's implementation, IBM, Sun's. . . . We've got every major vendor that will put us in a competitive place moving forward."
Visa is also a Cisco Catalyst switch shop that doesn't yet require the 1.7Tbps capabilities of the Nexus 7000 switch. Indeed, Lewis cautioned that, with virtualization coming back into vogue thanks to newer players like Cisco, adopting a buzzy technology for technology's sake is not in the Visa blueprint.
"Virtualization does nothing for you unless you have the integrated processes, the organization, the commitments from various partners and vendors that you work with to address pricing methodologies, contracts," Lewis says. "There are a number of aspects that will lead toward our full data-center strategy. In itself, it does nothing without the other disciplines."