HPC User Site Census: Storage

HPC User Site Census: Storage

This report, part of our Site Census research, provides an examination of the storage characteristics and capacities found in a sample of High Performance Computing (HPC) user sites. Intersect360 Research surveyed a broad range of users about their current computer system installations, storage systems, networks, middleware, and supporting software. The resulting data is presented in a series of reports throughout the year, each describing different aspects of the HPC end-user landscape. This report examines storage usage within the HPC user community and explores how this usage varies based on categories such as storage capacity, site configuration, supplier, and network.

Key findings of the survey include the following:

Storage capacity continues its onward and upward trend. Total storage capacity at over 400 sites exceeded three exabytes, triple the total storage capacity reported in 2012. While petabyte storage system installations are increasing, large storage systems only account for 9% of all storage systems reported. 50 TB to 100 TB storage systems account for the largest segment, with 30% share.

SSDs are increasingly being used for local node storage, with 20% of the systems last modified in 2013+ reporting at least some SSDs being used for local node storage. Very few of the systems with SSDs reported 100% usage, which suggests that most SSDs are being used as an additional tier between memory and traditional hard disks in order to improve storage-to-memory latencies.

Approximately 69% of storage (excluding outliers) is found at the site level. No single vendor dominates the storage system market for HPC sites. DataDirect Networks (DDN) has the largest share with 14.4% of survey responses, and is closely followed by IBM with 14.2%.

Parallel file systems account for 44% of the 342 named storage management packages, which is about the same as last year. Most parallel file system usage was found in storage systems with capacities of 200 TB or more and in storage systems last modified in 2013+. Most storage management software in use by the surveyed sites (59%) was provided by the storage system vendor. GPFS and Lustre continue to be most frequently mentioned named storage management packages with 16% and 15% of the systems, respectively.

Ethernet, and in particular, 10 Gigabit Ethernet, was the network protocol for almost 60% of storage systems. InfiniBand, however, captured more installations for storage systems installed in 2013+ with 47% share, suggesting successful competition against the higher speeds of Ethernet (10G, 40G, and 100G).

Storage Hierarchy Capacity
Node-Level Disk Capacity
Shift to Larger Disks per Node
Use of Diskless Node Declines
SSD Use for Node-Level Disk
Node-Level Storage by Interconnect
Node-Level Storage by Node Count
Node- and System-Level Storage Capacity by System Architecture
Node and Storage Outliers Excluded
Site-Level Storage System Analysis
Storage System Characteristics
Site-Level Storage Systems by Supplier
Cloud Storage Suppliers
Site-Level Systems by Primary Network
Primary Networks by Sector
Primary Networks by Budget
Site-Level Storage Capacity by Primary Network
Storage Management Software
Parallel File Systems
No Vendor Dominates
Technology Transitions
Guidance to Vendors
Appendix A: Complete Listing of Storage System Suppliers Reported
Appendix B: Complete Listing of Storage Management Software Packages Reported

Download our eBook: How to Succeed Using Market Research

Learn how to effectively navigate the market research process to help guide your organization on the journey to success.

Download eBook