This post started as a simple note on an issue in a couple of client’s implementations and has grown from there. It is not a complete and thorough covering, but it does have some starting points.
I’ve done a couple of implementation reviews recently where poor VMFS datastore design was causing real headaches for the customers. Typically they are customers who have implemented a large number of small VMFS datastores, nearly a datastore for each VM.
The aim of VMFS datastores is to be able to store a number of vmdk files in a single container. VMFS performs well with a moderate number of vmdk files in it, VMware suggest around 30 vmdk files per VMFS as a threshold, however the VMFS file system allows for over 30K files. This allows the consolidation of free space, i.e. minimises the chance of filling up the datastore. Separate datastores should be maintained for production and test/development environments and different datastores will be required to achieve different storage performance.
Below are a few common questions:
How large should I create my datastores?
The best answer is start with 1/2 TB and change from that as required. This suits having OS disks on a datastore as 16GB x 30 vmdk files is around 1/2 TB. 16GB is a good size for the system disk of a Windows Server so a single VMFS could happily hold the system disk for 30 Windows VMs. Naturally the VMs would have requirements for data disks, these may suit being on the same VMFS as the OS disk (leading to fewer VMs on the VMFS to stay around 30 vmdk files) or they may suit being on a different VMFS that has a different physical storage configuration.
So when should I have a VMFS of a different size?
The first reason would be if you need larger vmdk files, such as file servers with large storage requirements, you will need a datastore of sufficient size to hold the vmdk files required. The second is if you don’t need a datastore 1/2TB in size, maybe because you don’t have a lot of vmdk files to put on a single datastore, which leads to the next question.
What should I put on separate datastores?
1. The first reason to have things on separate datastores is to alleviate VMFS locking.
When a file is created or deleted from a datastore or the size of a file changes the VMFS metadata must change, to reflect the changed block assignment. The metadata change requires a lock on the whole of the VMFS. While the change is being written to the VMFS nothing else is able to read or write the datastore. As you can imagine if this happens a lot the performance of the VMFS will suffer. the prime culprit is snapshots where the redo log, which holds the up to date contents of changed vmdk disk blocks, will grow over time. If there are a lot of redo files and/or they are changing size fast then performance suffers.
Keep dev/test systems where there will be a lot of snapshots, on a different VMFS datastore from Production systems.
2. The second reason is that different datastores can have different storage characteristics.
One VMFS could reside on some tier one Fibre Channel disks which are extremely fast and expensive, while another could reside on a SAS array and another on a SATA array for progressively slower and cheaper storage. Different VMs, even different vmdk files for the same VM, could live on different speed and cost of disks. In the same way you could have a range of FC, iSCSI and NFS based datastores with varying performance, cost and other features.
3. The third is a bit of a repeat, VMware suggest 30 vmdk files per Datastore.
The recommendation may be out of date, but I like the idea of a boundary for the maximum number of disk files that can be lost by deleting one partition table (or removing one datastore as customers have been known to do)
What are the hard limits?
The best place to find the limits on any part of ESX is to look at the Configuration Maximums guide.
The key limits are 2TB per LUN, 2TB per vmdk (requires VMFS datastore with 8MB block size) and 64TB per VMFS Datastore (minimum 2MB VMFS block).
Wrapping it up
Overall there is a balance to be struck between few large datastore and more smaller datastores, if you have less than a hundred VMs (as my clients did) you definitely shouldn’t have dozens of Datastores.
© 2009, Alastair. All rights reserved.