The Department of Energy has awarded a $325 million contract to IBM to create two supercomputers that will be at least three times more powerful than any existing systems in deployment today. IBM's partners in this endeavor will be Nvidia and Mellanox.
The current leader is Tianhe-2 (Milky Way 2), a Chinese supercomputer with a theoretical max of 55 petaflops built with Xeon E5 processors and Xeon Phi co-processors. It may or may not be surpassed when the new Top500 supercomputer list comes out this week. Either way, a 165 petaFLOP supercomputer is a tall order.
The DoE supercomputer will use a mix of IBM Power 8 RISC CPUs, Nvidia's Tesla GPUs and NVlink GPU interconnects, and Mellanox's 100Gbit/sec. InfiniBand interconnects. The system is expected to be installed in 2017.
The timing is perfect. Last month, IBM announced the OpenPOWER Foundation, its effort to combine Power processors with Nvidia GPUs as co-processors for chewing through massive data sets. At the time, IBM announced a few systems it felt had a power/performance advantage over Xeon-based servers, and has tuned DB2 to take advantage of the performance gains, which will likely give Oracle fits. So at least IBM and Nvidia aren't starting from scratch with this DoE project.
This is a first. Before this, Nvidia GPUs have never been used in IBM Power systems. If you look over the Top500 list, there are plenty of IBM systems running Power processors but with no Nvidia or any other GPU. And there are plenty of Nvidia-powered systems on the list, but they are in x86-based servers.
The computers will be based at the Oak Ridge National Laboratory and Lawrence Livermore National Laboratory, already the sites of massive supercomputers. The scale of these systems is mind-blowing; both will have more than five petabytes of dynamic and flash memory and will be able to move data to the processor at more than 17 petabytes per second.
The unsung hero in these two systems may be Nvidia's new NVLINK technology, an ultra-high-speed interconnect capable of transferring data at a rate of up to 200 gigabytes per second. It will allow the CPU and GPU to communicate five to 12 times faster than standard PCI Express Gen3 interconnects, according to Nvidia. NVLINK uses 3D stacked memory that increases bandwidth by four-fold, increases capacity by two to three times old memory capacity, and is four times as energy efficient as regular memory.
IBM and Nvidia are united against a common enemy – Intel. The two both compete with the firm, and given Intel's dominance of the Top500 list, both IBM and Nvidia have reason to link arms. There are a lot of promises in the IBM press release. Let's see if they can keep them, because that's a lot of tax dollars.