Demitasse

Today I achieved a new trainer status, I am now a Microsoft Certified Trainer.  With Microsoft entering the hypervisor based virtualisation market it’s a god time to get into teaching their material as a compliment to the VMware courses I teach

You should by now know that I like unattended installs of ESX, largely because I know that I don’t follow instructions, even the ones I write.

One of the labs in the VI3  DSA course is to do an unattended install, it used to be the first lb & now it’s the last lab.  Last week as I was teaching the DSA in Wellington NZ my students had a little problem with their unattended installs, they were unable to download the kickstart unattend file.   It turns out that the DHCP server in the lab was not behaving as we’d like.  After a quick Google round and found an article on Tradmill’s blog which lead me to the little nugget of how to pass the IP address instead of using DHCP.  Then the line on the grub screen changed from:

esx ks=ftp://192.168.150.1/ks/esx1.ks method=ftp://192.168.150.1/ESX350 /ksdevice=eth0

to:

esx ks=ftp://192.168.150.1/ks/esx1.ks method=ftp://192.168.150.1/ESX350 /ksdevice=eth0     ip=192.168.150.2:192.168.150.1:192.168.150.1:255.255.255.0

Sure it ain’t pretty but it does work. 

The ip=w.w.w.w:x.x.x.x:y.y.y.y:z.z.z.z looks like this:

w.w.w.w is the IP address for the local NIC

x.x.x.x is the IP address of the remote server to connect to

y.y.y.y. is the IP address of the default gateway

z.z.z.z is the subnet mask

This got the lab running and students were able to build their ESX servers without touching the keyboard again.

This week I’m working with a customer whose blade based ESX servers are in a datacentre without DHCP, so I’ve setup an iso file that builds the servers.  My isolinux.cfg file snippet changed from:

label esx
  kernel vmlinuz
  append initrd=initrd.img ks=ftp://192.168.150.1/ks/esx1.ks method=ftp://192.168.150.1/ESX350 /ksdevice=eth0

to:

label esx
  kernel vmlinuz
  append initrd=initrd.img ks=ftp://192.168.150.1/ks/esx1.ks method=ftp://192.168.150.1/ESX350 /ksdevice=eth0     ip=192.168.150.2:192.168.150.1:192.168.150.1:255.255.255.0

I’ll post about the iso image and how ESX 3.5 has made that a whole lot easier in a future entry.

A much discussed feature is acually present in the VI Client, that is the ability to have the local userID and password automatically used to authenticate against the VC server.  According to this post it is as simple as changing the shortcut to add the directives “-passthroughAuth -s vchostname” to the shortcut. 

It appears that this is not a supported feature, but experimental is good enough for me to play with it.

I was worried about the health of my VMs after I upgraded (rebuilt, not an in-place, never in-place upgrade) to ESX3.5 and VC 2.5.

My VC VM was using 100% cpu and my VI client connections were timing out, also my SBS server VM was sluggish.

After a while I Googled 100% CPU and VC 2.5 and ended up with the VMwareWolf, his post told me just what was wrong and how to fix it.  After applying the vpxd.cfg fix listed at the end of the VMware KB article my VMs are running well and I’m ready to start teaching the VI3.5 version of the Install and Config course.

• Good news, this issue is resolved in VC 2.5 Update 1

I found a weakness in my use of a windows server to my NFS datastores when I went to migrate my SBS server to the NFS datastore as a part of my VI3.5 upgrade.

In an earlier post (Windows NFS with VI3) I linked to directions for setting up Microsoft Services for Unix too allow a Windows server to host an NFS datastore.  I’ve used this to host low priority VMs and all of my ISO’s and templates and am really happy with it.  I planned to use the NFS datastore as a staging point to get my SBS server VM from local storage in my ESX 3.0 server to shared Fibre Channel storage on my ESX 3.5 servers.

I used Mike Lavericks Free RTFM Guide: “What’s New & Different in Vi3.5” to look at Storage VMotion and moved a few VMs from local storage to the NFS datastore.  Then I tried to move my SBS server, this started but never finished.  Rather worryingly it left me with all sorts of odd VM’s in VC.  After spending a while untangling the vmdk and vmx files that were created in the process I got my SBS server back in operation and on the original storage.

Then I started to think through what might have gone wrong.  In the end I decided it must have been the NFS datastore requiring authentication at the wrong moment in the SVMotion.  I looked at the NFS config and found two things:

1. I had used a domain admin account to map to the Linux root userID, this would require a domain controller for authentication.  Since my SBS server was my only DC if it was off the air even briefly the authentication would fail.
2. I had accepted the default of “Renew authentication every:  600 seconds” in the “Server for NFS” “Server Settings” dialogue.  This would require authentication to be refreshed every 10 minutes for the NFS datastore.

The moral of the story is that when the blog posting about NFS says use a local administrator account for the mapping then you should.

Now I use a local account for the mapping, I don’t require authentication to renew and I have a backup domain controller.

Even better my production VMs are all on a proper SAN.

The VI3 Install & Config course has a slide which talks about setting up the Service Console networking for HA.  The key design item is to ensure that the SC network is fault tolerant but is still a representation of the health of the VM network.

The HA cluster is designed to start up VMs that were running on an ESX server in the cluster if that server has failed catastrophically.  If the ESX server crashes then HA does a great job of powering on those VMs (let’s not get into the survivability of the guest OS on random power cycling, that’s dependent upon your guest OS).  The place where we need to get concerned is where there is a partial network failure.

HA does it’s job by monitoring network communications between the Service Console on each ESX server in the cluster.  If an ESX server can talk to some nodes in the cluster but not others then after 12 seconds of lost communication each ESX server attempts to ping an IP address, by default the default gateway for the SC.  If this IP address is pingable by an ESX server then this server is part of the survivors of the cluster and the ESX servers that are uncontactable is failed.  If this IP address is not pingable then this server is isolated from the cluster and should do what it can to allow the surviving nodes to power on the VMs it is running.

Consequently the SC networking is mission critical to VMware HA.  If the SC networking fails then HA believes that there is a fault in the cluster.  So HA is implicitly monitoring network connectivity as well as SC “upness”. 

Consider the simple setup below, VM network separated from SC network with separate vSwitches:

If vSwitch0 looses connectivity with Physical Switch 0 the ESX server is deemed to be isolated, despite their being no impact on VM availability.  However if the connections to Physical Switch 1 fail the VMs are unavailable but no HA activity will occur.  Clearly HA is not monitoring the right network.

So how about this one, VM network separated from SC network with different VLANs:

Here SC and VM traffic are both carried over the same physical links, so if SC has lost it’s physical links then so has the VM.  However (that’s but with more letters) we’re now dependent upon the SC VLAN being up and I believe that Cisco trains network engineers to Shutdown a VLAN before making changes to it.

I have definitely heard of environments that used the above configuration and had every VM in their whole HA environment shutdown because the networks team were making VLAN changes.

The best solution I can find is to use a single vSwitch with VLANs to segregate traffic (and maybe NIC active/Standby set to guaranteed bandwidth) and at least two SC ports on different VLANs, like the diagram below:

Here HA monitors the physical networks used by VMs but the risk around VLAN shutdown is reduced as two VLANs must be concurrently shutdown to cause an HA event.

Please suggest any alternate configurations

In the first part of this series I looked at using the new Pocket ACE feature of VMware workstation 6 and the ACE add-on.  My conclusion was that the install process needed work.

The next tool I have looked at and had fun with is the Moka5 engine.  This is a user friendly wrapper for both installing the VMware Player and for getting the VM onto a PC or USB key to run it from.

In the role of USB based VM environment to be used on a PC that I’m visiting this is much slicker than Pocket ACE.  The Player install is silent, although I expect the same limitations for true kiosk PCs where MSI packages are locked out.

Last week I was a student on a course so I used a classroom PC for four days, the VM I used for the ACE work gave me what I needed in the VM for the week.

The hard part was getting my VM into the setup.  The Moka5 engine is designed to allow you to have a library of VMs on their hosted web site & download the VM or parts of VM that you require to whatever install of Moka5 you happen to be using.  You can have a private library of VMs or access the public library.  The public library is all free software at this stage, although there is an option to pre-order Windows XP & Vista VMs.  The VM libraries are simply web folders, so it would be easy to implement a corporate library.

The Moka5 engine also lets you import existing VMs from VMware Workstation.  I had initial difficulties with this as I had created a pre-allocated disk on my VM and at one point it was a SCSI disk.  Both of these choices caused the Moka5 import to fail.  Once I had a VM with IDE disks and sparse provisioned the import behaved correctly.

The other interesting feature that the Moka5 engine has is a Bare-Metal install mode, i.e. install to bootable media.  I tried this with a USB key and it worked well, although since I haven’t setup a library of MOKA5 Live PCs (or LPCs as they like to name their packaged VMs) I couldn’t test this a huge amount.  I did test downloading a couple of the public library LPCs and these ran well inside the Linux based boot environment on the USB key.  The Bare-Metal engine can be installed to a hard disk in a PC allowing the PC to be used as a host to run LPCs that are downloaded from a web site.

On a USB key it isn’t yet possible to use the same copy of a VM for both the Bare-Metal and host OS access methods, but this should be a development priority.

Right now Moka5 has a permanent place on my USB key, as a way to get to a large amount of my computing environment from a PC I don’t own.

I need to do some tuning work on the VM, particularly finding a web service to sync my favorites and maybe some sort of file sync for frequently used documents.

In the VI3 Install & Config class I recommend to students that they use a physical windows machine to hold a library of ISO images to install and update VMs from.  I recommend that they use Microsoft’s Services for Unix to make the same library available as an NFS datastore and a windows share.

It wasn’t until I came to set this up in my office that I found it wasn’t as simple and easy as you’d hope.  So naturally I turned to the great revealer for an answer.  The answer that I liked was in the form of a blog entry that outlined the steps very neatly.  The rest of the site is worth a look too, particularly the Shared Blogger part.

I can now go back to telling students to set up Services for Unix (or Services for NFS in Win2003R2) with a clear conscience.

VMware’s converter product does an excellent job of converting existing physical servers into Virtual machines.
However those who have read earliier posts will know it’s not perfect,some servers don’t P2V well, so what shouldn’t be P2Vd?
Domain Controllers
First off domain controllers, lots of people have had iossues with P2V of DCs, it’s a better idea to build a new DC in a VM and DCPromo it into the domain. If you must P2V a DC then you should DCPromo it out before the P2V,then after the P2V you can DCPromo it back in. This would occur if you had a DC that had a number of other roles.
Citrix / Terminal servers
You should have an automated build process for these (unless you only have one) so it should be easy to build a new one in a VM using that methodology.
Exchange Servers
An Exchange mailbox server shouldn’t be P2Vd as it is easier and more manageable to build a new server and then migrate mailboxes from the old to the new. P2V is probably fine if you only have a few mailboxes and consequently a small database, but a 2000 mailbox server with a huge database is going to take forever to image.