Why do we need storage with special sauce?

There seems to be a lot of companies with new storage solutions, lots of very smart people finding new ways to do make your storage better. Many of the startups are doing something very different to what the big storage vendors have been doing for years. So why are these startups making storage with special sauce?

What is the problem?

The fundamental problem is that hard disks have not been getting faster anywhere near as rapidly as they are getting larger.  Combine this with the habit of buying storage by the terabyte and you have the cause. Imagine you are a manager buying a storage system in 2003, you need ten terabytes of raw capacity so you buy around seventy disks, 146GB each. Those disks could each deliver about 150 transactions per second (IOPS), leading to a little over 10,000 IOPS from the seventy disks. Now fast forward to 2013, the same ten terabytes of capacity can be satisfied with ten disks, since they are each 1TB. The new disks can deliver 180 IOPS each, so 1,800 total for ten terabytes, less than a fifth of the IOPS that the same capacity would have delivered ten years before. To get the 10,000 IOPS with modern disks you would need to buy round fifty nine disks, nearly six times the capacity that was required before.

At the same time the storage workload has been concentrated, before virtualization most physical servers used local storage and a small proportion of physical servers accessed the SAN. Now that virtualization has taken hold and we have hundreds of VMs all stored on the SAN there is a lot more work for the SAN to do.  Also the VMs are rather harder on the SAN than most physical servers since the virtualization hosts mix multiple VM IO into each LUN, leading to very heavy workloads for the SAN. This also means that SANs need to cope with a lot more small and random IO than they did before virtualization

Quick and Dirty Comparison of Storage

It’s worth a quick look at types of data storage technology currently in use, there are basically three tiers:

Each tier has very different characteristics and needs to be treated differently, each tier can be placed inside a physical server or in a central array and can bring different value based on where it’s put. Hard disk is usually the final long term storage location, although we are seeing more solid state as the final location.

What are the solutions?

The fact that there are a lot of startups doing different things suggests that there are a few possible solutions, for different parts of the problem. Here are three basic types that address the IOPS/Latency demands:

1. Lots of disks and lots of cache in the storage array.  This is the solution for the existing storage vendors, do more of what they have done in the past without changing the architecture or the technologies.  This solution gets expensive, particularly since hard disks are power hogs and take up a lot of datacenter space.
2. Faster disks, solid state. Unlike a hard disk a solid state disk can easily be made to get faster as it gets larger, and since it isn’t rotating there is much lower latency to access data, this means more IOPS. This type of solution is relatively new and the “best” way to use solid state disks isn’t yet clear. One things that while it is clear is that treating solid state exactly the same as hard disk or RAM isn’t the best use of technology, it is often good enough to help customers.
3. Cache closer to compute. These solutions work with existing storage arrays and add a layer of caching in front of the storage array, usually closer to the virtualization hosts that are running the workloads. This is probably the newest approach, now that RAM or solid state storage in the virtualization hosts isn’t so expensive we’re seeing solutions pop up that use the lower latency of these devices to deliver faster storage response to VMs. Keeping the IO off the storage fabric and close to the applications (VMs) that are generating the IO reduces the strain on the SAN.

There is definitely a need for special sauce in storage, adding more spindles of hard disk just doesn’t scale to the requirements of virtualized environments. There are a heap of other storage special sauces and a whole lot of ways that startups and the big boys are approaching a lot of other storage challenges, like snapshots, replication  and deduplication, but that will need to be another post or two.

© 2013, Alastair. All rights reserved.