Why do we need storage with special sauce?

There seems to be a lot of companies with new storage solutions, lots of very smart people finding new ways to do make your storage better. Many of the startups are doing something very different to what the big storage vendors have been doing for years. So why are these startups making storage with special sauce?

What is the problem?

The fundamental problem is that hard disks have not been getting faster anywhere near as rapidly as they are getting larger.  Combine this with the habit of buying storage by the terabyte and you have the cause. Imagine you are a manager buying a storage system in 2003, you need ten terabytes of raw capacity so you buy around seventy disks, 146GB each. Those disks could each deliver about 150 transactions per second (IOPS), leading to a little over 10,000 IOPS from the seventy disks. Now fast forward to 2013, the same ten terabytes of capacity can be satisfied with ten disks, since they are each 1TB. The new disks can deliver 180 IOPS each, so 1,800 total for ten terabytes, less than a fifth of the IOPS that the same capacity would have delivered ten years before. To get the 10,000 IOPS with modern disks you would need to buy round fifty nine disks, nearly six times the capacity that was required before.

At the same time the storage workload has been concentrated, before virtualization most physical servers used local storage and a small proportion of physical servers accessed the SAN. Now that virtualization has taken hold and we have hundreds of VMs all stored on the SAN there is a lot more work for the SAN to do.  Also the VMs are rather harder on the SAN than most physical servers since the virtualization hosts mix multiple VM IO into each LUN, leading to very heavy workloads for the SAN. This also means that SANs need to cope with a lot more small and random IO than they did before virtualization

Quick and Dirty Comparison of Storage

It’s worth a quick look at types of data storage technology currently in use, there are basically three tiers:

  Hard Disk Solid State RAM
Technology Spinning plates of magnetic media Integrated circuit Integrated circuit
Max unit size 4,000GB 1,000GB 32GB
Max unit IOPS 200 20,000 500,000
Access latency 10 Milliseconds 100 Microseconds 70 Nanoseconds
Data persistence when powered off Yes Yes No
Cost per TB Low to medium High Very high
Strengths Lots of capacity Fast, low latency Very fast, very low latency
Challenges Slow, physically large and power hungry Small, expensive, relatively new Very small, very expensive,
looses content when powered off

Each tier has very different characteristics and needs to be treated differently, each tier can be placed inside a physical server or in a central array and can bring different value based on where it’s put. Hard disk is usually the final long term storage location, although we are seeing more solid state as the final location.

What are the solutions?

The fact that there are a lot of startups doing different things suggests that there are a few possible solutions, for different parts of the problem. Here are three basic types that address the IOPS/Latency demands:

  1. Lots of disks and lots of cache in the storage array.  This is the solution for the existing storage vendors, do more of what they have done in the past without changing the architecture or the technologies.  This solution gets expensive, particularly since hard disks are power hogs and take up a lot of datacenter space.
  2. Faster disks, solid state. Unlike a hard disk a solid state disk can easily be made to get faster as it gets larger, and since it isn’t rotating there is much lower latency to access data, this means more IOPS. This type of solution is relatively new and the “best” way to use solid state disks isn’t yet clear. One things that while it is clear is that treating solid state exactly the same as hard disk or RAM isn’t the best use of technology, it is often good enough to help customers.
  3. Cache closer to compute. These solutions work with existing storage arrays and add a layer of caching in front of the storage array, usually closer to the virtualization hosts that are running the workloads. This is probably the newest approach, now that RAM or solid state storage in the virtualization hosts isn’t so expensive we’re seeing solutions pop up that use the lower latency of these devices to deliver faster storage response to VMs. Keeping the IO off the storage fabric and close to the applications (VMs) that are generating the IO reduces the strain on the SAN.

There is definitely a need for special sauce in storage, adding more spindles of hard disk just doesn’t scale to the requirements of virtualized environments. There are a heap of other storage special sauces and a whole lot of ways that startups and the big boys are approaching a lot of other storage challenges, like snapshots, replication  and deduplication, but that will need to be another post or two.

© 2013, Alastair. All rights reserved.

About Alastair

I am a professional geek, working in IT Infrastructure. Mostly I help to communicate and educate around the use of current technology and the direction of future technologies.
This entry was posted in General. Bookmark the permalink.

One Response to Why do we need storage with special sauce?

  1. Alex says:

    This is a great article, I think there are a couple of more things which underline this.

    If you are buying your storage from a “traditional” vendor, their “smallest” disks are probably almost same price their “smallest disks” were in 2003. So 10k IOPs probably don’t cost much less than they did in 2003 either !

    Bearing in mind the price difference between enterprise SSDs and consumer SSDs (making up numbers here – about 10x per GB/and about 5x per IOP). If you can do something clever in software to get enterprise endurance and performance out of consumer SSDs – you are onto a winner.

    Most SSDs once put in a RAID configuration to provide resilience and the required capacity, have *FAR* more IOPs than the workload requires. If you can do something to provide a blend of SSD performance and spinning disk capacity – you are onto a winner.

    Most workloads don’t have TB of data which are accessed completely randomly if at all. While there may be many TB of data in total in many companies – a high % of it is tends to be pretty cold. This is because archiving is expensive, and throwing disks at it is cheaper. If you can do something clever to make sure that hot data is on expensive fast disks, and old data is compressed on slow cheap disks – you are onto a real winner.

    Then when you weight up the fact that “traditional” vendors are desperate to defend their legacy revenue sources – and providing clever ways of selling a lot less disks, isn’t really that compatible with that. So they are forced to walk a very fine line of giving people what they want to keep customers – but not too much of what they want to not cannibalise their own revenue.

    The only really interesting product launched in this space, by a big vendor was the ZFS/fishworks storage appliance – and look what happened to them !

    So fair play to the special sauce vendors, and long may they cannibalise the big guy’s revenue.



Comments are closed.