The NCSA also selected a DataDirect Network's SFA 12K storage array delivering 100GBps of storage performance to offload data to the "nearline" tape library system. The disk subsystem is scalable to 500PB of capacity, Butler said.
"That subsystem has to be able to offload that terabyte-per-second file system, so we needed a very large tape drive infrastructure to be able to offload that," she said.
The Blue Waters supercomputer in the process of construction. (Photo courtesy of the NCSA.)
Behind the primary storage are four Spectra Logic 17-frame T-Finity tape libraries that will have 366, 240MB/sec. IBM TS1140 enterprise-class tape drives. The libraries will offer an aggregate read/write rate of up to 2.2PB per hour.
"We actually evaluated either LTO-5 or LTO-6 and the TS1140. We didn't specify what tape drives, what libraries or anything. We wanted the vendors to have the freedom to propose multiple solutions to us," Butler said.
Butler said the NCSA chose the IBM tape drives over more popular midrange LTO drives because they offer superior performance. The TS1140 offers 240MB/sec. throughput compared to about 140MB/sec. for the LTO drives, she said.
In its request for proposals, Butler's team highlighted 10 to 15 requirements for storage vendors to meet. Among other things, they stipulated that the tape library would have to fit into a certain square footage, could not exceed certain power and cooling requirements, and should meet certain reliability and performance targets.
Butler said the goal for aggregate throughput on the tape library was 100GB/sec. Currently, it's right around 89.5GB/sec.
The Cray supercomputer is connected to its tape library via Mellanox IS5000 InfiniBand switches and ConnectX InfiniBand adapters for its tape library storage network. The switches are using the InfiniBand QDR protocol, which offers up to 8Gbps throughput per lane and up to 12 I/O lanes. Butler said she wanted to use the higher bandwidth version of InfiniBand, FDR, but Cray's systems didn't support it.
InfiniBand FDR (Fourteen Data Rate), offers up to 13.6Gbps throughput per lane or 163.6Gbps over 12 I/O lanes.
While the NCSA could have chosen products from any number of enterprise-class disk storage vendors for use in the supercomputer, Butler and her team felt the NCSA would receive better support if it all came from Cray.
"Lustre, as you probably know, is not the easiest thing to take care of and maintain, so we wanted to partner with one specific [vendor] for the software and hardware and have an appliance to do the failover and that hard stuff. And, we've been running Lustre here since 2003," Butler said. "So I understand [Cray] trying to simplify our system for us."