The Future of Storage: Hardware
Fueled by massive growth in data creation, the state of data storage hardware is evolving at a considerable rate at the moment. Much of the innovation occurring today is tied to solid state flash drives and non-volatile memory (NVM) technologies, running both on-premise and in the cloud. But existing electro-mechanical storage media like tape and spinning disk have maintained a precarious foothold.
By all accounts, data is growing at an accelerating rate, and is now close to doubling in size every year. In EMC‘s 2011 “Digital Universe” report from 2014, the IT analyst group IDC pegged the yearly growth at 33% (i.e. doubling every three years), while the 2014 iteration of the report upped the growth rate to 40%. In 2017, IDC predicted a 10x increase in data by 2025, which translated into a 70% annual rate.
Much of the data growth can be tied to unstructured data. In 2016, IBM shared the results of an IDC report that claimed that, out of the12 zettabytes of data created in 2016, 80% of it was unstructured data, such as video, audio, Web, social media, email, and log and genomic files. IDC predicted the world would create 44ZB of all types of data in 2020, representing a 92% annual growth rate.
Another way to understand the state of data storage is to see what storage vendors are actually shipping. Last month, IDC released the results of its Worldwide Quarterly Enterprise Storage Systems Tracker, which found that total value of storage arrays shipped during the first quarter of 2018 totaled $13 billion, which was a 34.4% increase from the first quarter of 2017. Dell led the revenue race, followed by HPE, NetApp, and Hitachi and IBM tied for fourth. Total capacity amounted to nearly 99 exabytes, which was a 79.1% increase from a year ago.
As you can see from IDC’s numbers, we’re gobbling up enterprise data storage systems at an exceptional rate. Thanks to the continued generation of all types of data and the on-going decline in price-per-gigabyte (or price-per-exabyte, if you prefer), there’s no indication the growth will slow anytime soon.
What Storage People Are Buying
So, what kinds of storage systems are companies buying? According to IDC, the All Flash Array (AFA) segment – which refers to storage systems entirely composed of solid-state flash drives that speak NVMe — is red-hot at the moment.
Storage system makers shipped $2.1-billion worth of AFAs in the first quarter, which was a nearly 55% increase from the first quarter in 2017. Many of these AFAs ostensibly are going to supply data to transactional systems and relational databases, which are often well-suited to handle the greater I/O capacity that flash disks can provide.
By comparison, the market for Hybrid Flash Arrays (HFA), which combine SSDs and traditional spinning hard disks, generated $2.5 billion in revenue, which was up about 24% from 1Q17. While HFAs can’t offer the speed of AFAs, they can deliver some of the benefits without the higher costs. Customers that have deployed data lakes on Hadoop, for example, can see a substantial performance increase by adding a single $2,500 flash drive to each Hadoop node.
Who’s buying all this flash and non-flash storage? According to IDC, hyperscale datacenter operators (i.e. Amazon Web Services, Google Cloud, and Microsoft Azure) bought $3.1 billion of storage from original design manufacturers, or so-called “white box” storage makers who aren’t one of the big five mentioned above. This figure represented an 80% increase from a year ago, and accounted for nearly 24% of all enterprise storage system sales, which shows you how quickly the cloud is growing at the moment.
Sales of server-based storage systems generated $3.6 billion in revenue, which was a 34% increase, while external storage systems accounted for $6.3 billion in sales, a 19% increase. The key takeaway from this is that all types of enterprise storage are growing at the moment, but flash technologies are growing the fastest.
While flash storage technologies are hot at the moment, people and companies still rely and continue to buy a wide array of media to store data. Here’s a breakdown of popular storage media for big data:
HDD – Hard disk drives were first created by IBM back in 1956, and are still the workhorses of enterprise storage today, even if they’ve fallen out of favor thanks to the rise of flash storage. The largest HDDs today exceed 10 TB, such as Toshiba’s helium-filled MG07ACA Series disk, which can store 14 TB in a single 3.5-inch drive form-factor. Overall, HDDs are about 5x to 8x cheaper per TB than flash disks (see graphic), which at this point is the only advantage they have over flash.
Flash – Refers to a wide array of storage products that utilize NAND or other solid-state technologies to store data on memory-like modules as opposed to a magnetic or optical media. Flash drives have many benefits over HDDs, including faster data transfer rates, lower energy consumption, better mean time between failures (MTBF) rates, smaller physical sizes, and longer lifespans.
With the recent introduction of massive flash drives, like Nimbus Data’s 100TB ExaDrive, flash drives no longer play second fiddle to HDDs in the capacity department. The one downside to flash is they’re more expensive than HDDs — and thanks to high demand for flash chips from mobile device makers, the price difference isn’t getting better.
Tape – While HDDs and flash drives handle the hottest enterprise data sets, many companies still rely on massive tape libraries for backups and archives. The amount of data stored on tape is not trivial.
According to the LTO Consortium, LTO tape manufacturers shipped more than 100 exabytes of total compressed tape capacity in 2017, which was nearly a 13% increase from 2016. (That’s about one-fourth the amount of enterprise storage capacity tracked by IDC above, although the comparison narrows when you turn off compression.) The current generation of LTO technology, LTO-8, can store 12TB of data on a single cartridge, or 30TB with 2.5-to-one compression. While IBM makes LTO tape drives, Big Blue also conducts its own R&D for tape, which has yielded some breakthroughs such as its “sputtered” tape media introduced in 2017.
Optical – Once a go-to method for large enterprises that needed to archive data that must be readable for decades – including healthcare companies and financial services firms — optical storage techniques have largely fallen by the way side. However, owing to their capability to store data for 50 year or more, many companies continue to reliable and cost-effectively store petabytes worth of data on optical jukeboxes that were installed many years ago.
Flavors of Flash
While the term “flash” technically refers to just one type of solid-state storage technology – NAND — there are actually a host of related solid-state technologies competing to store tomorrow’s big data sets. They’re all collectively referred to as flash, but there are important differences.
Here’s a rundown on NAND and related technologies competing in the market:
NVMe –NVM Express (NVMe) has emerged as the go-to standard communications protocol for transmitting data at high speeds from memory to the CPU. Akin to SCSI or SATA, two communication protocols used for
HDDs, NVMe was defined by Intel, Samsung, Sandisk, Dell, and Seagate years ago and can take advantage of the PCI Express (PCIe) bus to move data at much higher speeds (with many more individual data pipelines) than other protocols. Flash drives that support NVMe have emerged as the defacto standard for solid-state storage, and have largely displaced flash drives that use SATA and SAS.
NAND – NAND (NOT-AND) is a persistent storage technology originally introduced by Toshiba in 1989 that implemented block storage on EEPROM (electrically erasable programmable read-only memory) sticks. Early flash cards were based on NAND technology, and it still forms the basis for many emerging storage products.
3D NAND – In three-dimensional NAND, memory cells are stacked vertically in multiple layers to address limitations in two-dimensional NAND cells and to boost storage densities. This is similar to Vertical NAND, or V-NAND. Intel is pursuing 3D NAND technology with its new Optane line of flash drives, which feature 64 layer memory chips.
Z-NAND – Introduced by Samsung in the fall of 2017, Z-NAND is said to deliver 10x the cell read performance of traditional V-NAND. It’s seen as a competitor to Intel’s Optane line of drives, which offered similar capacities and capabilities in shipping flash drives.
Phase Change Memory – Phase Change Memory (PCM), is a technology that competes with classic NAND-based flash technology. Also referred to as PCME, PRAM, or PCRAM, the optical technology offers faster writes than typical flash, and also doesn’t degrade as fast as flash as the writes add up over time. IBM and Intel are working on this technology, among others.
Micron are pursuing 3D Xpoint (pronounced “crosspoint”) as part of its Optane line of next-generation storage products. Originally announced in 2015, Intel and Micron are both shipping 3D XPoint products, which they claim offer 10x the density of DRAM and 1000x the speed and endurance of NAND media (although Intel and Micron recently split on the tech).
Storage Class Memory — 3D XPoint and Samsung’s Z-NAND are examples of so-called storage class memory (SCM), which can function either as storage or memory. There has been some controversy as to whether 3D XPoint technology is based on PCM technology.
The future looks bright when it comes to storage. While magnetic HDDs and tape still have a place, the overwhelming evidence says that newer flash drives based on NAND and other emerging technologies will promise to keep CPUs flush with big data to process for the foreseeable future.