Service Console Memory Starvation

I visited a client in November who was having a few issues with their Virtual Infrastructure.  One of the minor inconveniences they mentioned was that the VI client was timing out tasks a lot.  Sometimes this is due to WAN issues, however here the whole infrastructure was in one city.  It turns out that the issue was really Service Console memory starvation, to recognise SC starvation take a look at VMware KB article 1003496.

In the old days (ESX 2.x) there were a number of processes in the SC that were related to the VMs being run, consequently it was necessary to choose a SC RAM allocation dependent upon the number of VMs you were running.  In ESX 3.x these processes no longer exist so the SC RAM is set by the installer at 272MB, as that is sufficient to run as many VMs as an ESX server can run.

If you install agents or services into the SC you must increase the SC RAM to accommodate them.  The sorts of things people install into the SC include hardware management agents and backup applications.  Installing hardware agents is a good idea, far better to be notified of some correctable memory errors than have a Purple Screen of Death due to a non-correctable error a little later.  If you install any additional software in your ESX server SC increase the SC RAM to at least 400MB.

The client I visited had correctly the HP Management agents for their Proliant’s but hadn’t seen in the readme that they needed to increase the SC RAM to 400MB.  On a subsequent visit, after the SC RAM was upgraded the fault command timeouts were gone and performance data was being collected consistently.

Changing the Service Console RAM Manually

In the VI Client, on the Configuration tab select Memory and click Properties, set a new value and restart the host to apply.  For more detail look at VMware KB article 1003501.

Changing the Service Console RAM using a script

There are two options for the script from this VMware community post:

Either one line using the undocumented vimsh command, which  would be my preference

vimsh -ne “hostsvc/memoryinfo 419430400”

Alternatively you could use sed to directly manipulate the esx.conf file and then regenerate the initrd image, this method has apparently been authorised by VMware.

sed -i ‘s/memSize = \”272\”/memSize = \”800\”/g’ /etc/vmware/esx.conf
esxcfg-boot -g
esxcfg-boot -b

Other things to change

If you are installing software in the SC it is also important to make provision for the files the software will install and to ensure any log files are rotated correctly.  The default disk partitioning is adequate for most ESX servers where no additional software is installed.  You might want to increase the logs partition size and add partitions for either /opt or /usr to accommodate the software you install.  VMware have log rotation jobs for the built in logs and good software vendors include log rotation for their software, but it pays to be watchful.

© 2009, Alastair. All rights reserved.