Building a Test Workload

I’m working on a fun little project while I’m home for a few weeks. For a new vBrownBag project. I need to create a VM based workload generator. What I want is a collection of VMs that generate some significant workload for a virtualized infrastructure. I also want to be able to have different workloads that exercise the infrastructure in different ways. I have a few design criteria for both the workload itself and the method of deploying the workload.

Each workload needs to:

1. Use a real application type, not a purely synthetic workload
2. Drive multiple VM resource types, including network
3. Drive multiple VMs, ideally with interconnected load
4. Be scalable, both number of workers and rate of work
5. Have a random element, just like real users
6. Be largely autonomous in operation
7. Work independently of the hypervisor
8. Have a pod architecture, where multiple identical pods can operate concurrently

Deployment needs to:

1. Be as automated as possible
2. Allow complete rebuild in a short time frame
3. Work independently of the hypervisor
4. Work without an Internet connection

Deployment

At the moment, I am comfortable with having one unique VM as the center of the deployment. I’m calling this VM master and allowing it to have a non-automated build. I would like this’s build VM to be automated too, but that is a whole heap more work. The master VM serves as the repository for all the other builds and is the log aggregator for the running workloads.  This master VM serves as a TFTP and web server for automated builds of the VMs in the workload pods. I chose Ubuntu 16.04 LTS as the operating system for both the master and all the workloads. This was an entirely arbitrary choice but is working well. I also chose to use Ansible for most the build automation. I like Ansible as it has few dependencies for getting started. I could probably have used Chef or Puppet just as well, again an arbitrary decision.

Networking

To make the pod architecture work I need to be able to isolate the pods and re-use the same IP range for a collection of pods. I chose pfSense as the router/firewall between the pods. Each workload pod has its own pfSense router which provides outbound NAT as well as a small amount of inbound port forwarding. I also isolate the master VM in its own pod, with its own pfSense to isolate it from whatever else is on the network. This level of network isolation makes it easier to deploy the workloads alongside other workloads, like my lab.

Email Workload

The first workload I am working on is an email system.  Each pod has a server VM, this VM runs the mail server and is a central repository for the other VMs. Only the server VM is accessible through the pod pfSense, so I only use Ansible on the master VM to directly configure the file server. I then have Ansible on the file server configure the pod worker VMs. The pod worker VMs run a python script which downloads some input files of the file server and then sends emails. Each iteration through the workload script sends a single email address and uses the input files to randomize the email content. This workload is mostly IO driven, lots of network and disk IO for the emailing. There isn’t a huge RAM or CPU demand, apart from on the web/mail server. I want to add a database driven workload and probably a media transcoding workload to exercise different resource types.

Future Posts

In future blog posts, I plan to dig into the build automation, both the PXE setup and the Ansible parts. I may also show a bit more of the workings of the email workload and my decision to use Docker containers to run the workload. I haven’t yet decided whether I’m going to open source the whole kit and caboodle. I guess that I need to automate the master server build before I consider it usable by anyone but me.

© 2016, Alastair. All rights reserved.