This post started as a simple note on an issue in a couple of client’s implementations and has grown from there. It is not a complete and thorough covering, but it does have some starting points.
I’ve done a couple of implementation reviews recently where poor VMFS datastore design was causing real headaches for the customers. Typically they are customers who have implemented a large number of small VMFS datastores, nearly a datastore for each VM.
The aim of VMFS datastores is to be able to store a number of vmdk files in a single container. VMFS performs well with a moderate number of vmdk files in it, VMware suggest around 30 vmdk files per VMFS as a threshold, however the VMFS file system allows for over 30K files. This allows the consolidation of free space, i.e. minimises the chance of filling up the datastore. Separate datastores should be maintained for production and test/development environments and different datastores will be required to achieve different storage performance.
Below are a few common questions:
How large should I create my datastores?
The best answer is start with 1/2 TB and change from that as required. This suits having OS disks on a datastore as 16GB x 30 vmdk files is around 1/2 TB. 16GB is a good size for the system disk of a Windows Server so a single VMFS could happily hold the system disk for 30 Windows VMs. Naturally the VMs would have requirements for data disks, these may suit being on the same VMFS as the OS disk (leading to fewer VMs on the VMFS to stay around 30 vmdk files) or they may suit being on a different VMFS that has a different physical storage configuration.
So when should I have a VMFS of a different size?
The first reason would be if you need larger vmdk files, such as file servers with large storage requirements, you will need a datastore of sufficient size to hold the vmdk files required. The second is if you don’t need a datastore 1/2TB in size, maybe because you don’t have a lot of vmdk files to put on a single datastore, which leads to the next question.
What should I put on separate datastores?
1. The first reason to have things on separate datastores is to alleviate VMFS locking.
When a file is created or deleted from a datastore or the size of a file changes the VMFS metadata must change, to reflect the changed block assignment. The metadata change requires a lock on the whole of the VMFS. While the change is being written to the VMFS nothing else is able to read or write the datastore. As you can imagine if this happens a lot the performance of the VMFS will suffer. the prime culprit is snapshots where the redo log, which holds the up to date contents of changed vmdk disk blocks, will grow over time. If there are a lot of redo files and/or they are changing size fast then performance suffers.
Keep dev/test systems where there will be a lot of snapshots, on a different VMFS datastore from Production systems.
2. The second reason is that different datastores can have different storage characteristics.
One VMFS could reside on some tier one Fibre Channel disks which are extremely fast and expensive, while another could reside on a SAS array and another on a SATA array for progressively slower and cheaper storage. Different VMs, even different vmdk files for the same VM, could live on different speed and cost of disks. In the same way you could have a range of FC, iSCSI and NFS based datastores with varying performance, cost and other features.
3. The third is a bit of a repeat, VMware suggest 30 vmdk files per Datastore.
The recommendation may be out of date, but I like the idea of a boundary for the maximum number of disk files that can be lost by deleting one partition table (or removing one datastore as customers have been known to do)
What are the hard limits?
The best place to find the limits on any part of ESX is to look at the Configuration Maximums guide.
The key limits are 2TB per LUN, 2TB per vmdk (requires VMFS datastore with 8MB block size) and 64TB per VMFS Datastore (minimum 2MB VMFS block).
Wrapping it up
Overall there is a balance to be struck between few large datastore and more smaller datastores, if you have less than a hundred VMs (as my clients did) you definitely shouldn’t have dozens of Datastores.
© 2009, Alastair. All rights reserved.
RSS - Posts
Hi Alistair; something else you might want to mention; the defaults for snapshot location. By default the snapshots will be placed in the same place as the config files for your VMs. Example I’ve seen; several smaller datastores, a small-ish one of which holds the config files and all the OS drives for the dozen or so VMs that exist. Tech comes in to do Windows Updates across all servers on the same night (ok, so that strategy is another discussion!). Snapshots all the servers, leaves the snapshots ‘for a few days to make sure everything is ok’. Snapshots grow over the next couple of days, LUN fills up, no more free space = OH-OH…
After we soretd the problem out we had a chat with the tech concerned. Shouldn’t happen again!
Lesson: consider snapshot placement when designing your datastores. (you can, of course, jiggery-pokery VMs to change the snapshot location to an alternate location, I believe)
UPDATE: after chatting with Alastair by email, we both agree that leaving snapshots for ‘a few days’ is probably not the best plan for production systems. Reverted back to a several days old snapshot might, or might not, be the best idea!