Building a Nested ESXi Lab on VMware Workstation

If you are studying for the VCAP5-DCA you definitely need a lab. If you are studying for the VCP probably need a lab unless you are in vCenter all day at work. Nested virtualization runs one hypervisor upon another; so a nested ESXi lab runs the ESXi hypervisor on another hypervisor like VMware Workstation. So why build a nested lab instead of a physical lab?

Flexibility - A nested lab on workstation is going to provide more flexibility than a physical lab.  I have both, and I love having the ability to create another ESXi host in minutes by cloning it from a template. I can also turn off my 5.0 lab I am using to study for the VCAP5-DCA and turn on my 5.5 lab and show a coworker a new feature.

Cost - The cost of a nested lab can be cheaper than the cost of a physical lab, especially if you have box that you can simply upgrade the RAM in. When building a computer to run a nested lab the cost could be similar or more than buying used servers from eBay, but the power consumption should be much less. Building a low power solution like Intel NUC or MAC Mini's combined with a Synology will cost more than building a nested lab.

Portability - A small nested lab can run on a laptop allowing you to study on the road.

What do I need to build a nested lab?

Computer - One that supports VT-x (or the AMD version). Preferably one that supports EPT; without EPT support you will be limited to running 32bit guest virtual machines inside your nested ESXi instances. The 32bit restriction isn't a big deal, but it would be nice to not have to deal with it. If you are unsure of the virtualization features of your processor you can look it up at the Intel or AMD site. You should be aware that these features may not be on by default, you will need to check in the BIOS. 

RAM - Lot's of RAM. Did I mention RAM? Can you afford any more RAM? With ESXi 5.0 8GB of ram would allow you to get two ESXi hosts, vCenter, and an openfiler running. ESXi 5.5 brings higher minimum RAM requirements with all of the new features, 16GB really becomes the new minimum for two hosts, a vCenter, and an openfiler. If you want to lab larger scenarios like SRM or NSX you will need 32GB and up. 

I have a a Dell Precision T7500 Workstation with 48GB of RAM I jumped on when an engineer from our HPC group upgraded to newer model. It has an older processor, the Intel Xeon E5507, but it is quad core and supports VT-x with EPT so it meets my needs.  

VMware Workstation - Fusion will work as well, but I like the interface and memory overcomitment of Workstation. If you have your VCP they were providing workstation license keys upon passing, I'm not 100% sure if they still are. If you are a VMUG Advantage subscriber one of the benefits is a discount on the Workstation license. 

Dive into configuration after the break.

How do I build a nested lab?

1. Prepare the computer

Turn on the virtualization enhancements like VT-d and EPT in the bios. Install a 64bit operating system. I have only used Workstation for Windows but a Linux version is offered as well. Procure and install VMware Workstation; I've used both version 9 and version 10 with success for nested lab environments. The install is simple but you should pay attention to where it wants to keep your virtual machines. I normally change the default to "C:\Virtual Machines" so I know I can find them easily.

Now is a good time to edit the VM networking and plan out the "Host Only" and "NAT" networks that will be used in your environment, and customize the IP ranges if desired. I use VMnet9 for my storage traffic, VMnet7 for MGMT traffic, and the VMnet8 NAT network for guest VMs. You can customize the IP ranges here, just make sure you use ranges that aren't in use on your network. Your workstation will loose access to the network resources if it sees the range attached to a local interface. 

2. Create VM templates for linked cloning

 Much of the flexibility of using a nested environment for your lab comes from Workstation's ability to create linked clones from templates. This will speed your deployment of new scenarios and conserve disk space in the process. I currently have an ESXi 5.0 template, a 5.5 template, and a Windows Server 2012 template. I need to go ahead and deploy a linux template at some point in the future. The basic process it to customize the VM, generalize the VM, then power it off and make the VM into a template. 

ESXi Templates - To create a rapidly deployable ESXi template we need to create the virtual machine and deploy ESXi as normal, then generalize the hypervisor and shut it down. I only give the host a small 5GB disk for ESXi, a larger disk will result in creating a local VMFS datastore which can interfere with the cloning process. When creating the VM make sure that the "virtualize intel vt-x/ept or amd-v/rvi" option is checked.

Install ESXi from the .iso like normal, then from the DCUI generalize the template by unselecting the  vmnic0 for Management, clear the DNS suffix, and reset the system configuration. Power the host off before it reboots and it's ready to use as a template.

This procedure should work on 5.0 or 5.5, and there is a detailed guide to preparing ESXi templates by Josep Piscaer here:

http://www.virtuallifestyle.nl/2012/01/creating-a-vesxi-template/

You normally need more than one NIC in ESXi, I like to use six. Instead of putting all six in the template I start out with one then add them one at a time so I can keep them straight. If it starts up with six NICs sometimes it can be tough to determine which is which without upstream CDP. 

Windows Server Template : The main template I use other than ESXi is a Windows Server template. I run vCenter on windows instead of the appliance for the ease of integrating VUM. I'll probably want a domain controller to test AD integration. 

The general process to create a windows server template it to install the operating system and update it, customize it, and then generalize it with the sysprep.exe utility. Sysprep strips unique information from the machine and forces a small installer on startup. This process can be as complicated and automated as you want, I normally keep it simple for the lab machines.

Set sysprep.exe to shutdown the machine upon completion. Find a more detailed guide here:

http://willcode4foodblog.wordpress.com/2013/03/20/creating-a-server-2012-golden-image-with-sysprep-and-vmware-workstation/

Convert VMs to Templates : Once the ESXi and 2012 Windows Server virtual machines are customized, generalized, and powered off they can be converted to templates. The first step is to edit the settings of the VM, go to the options tab, advanced, and check the template option.

The second step is to create a snapshot that will be used to create the nested clones of our template. Right click on the template VM, snapshot, then create snapshot. 

3. Build the "Shared Infrastructure" VMs

I like to use the folder structure of Workstation to organize my lab. The first thing I create is a "Shared Infrastructure" folder that contains VMs that I want to stay powered up no matter which lab scenario I'm working on. I keep an openfiler VM here to provide NFS and iSCSI datastores to the ESXi hosts, and an AD domain controller to test AD integration.

Openfiler : ESXi labs need some type of storage. I like to have both NFS and iSCSI available. Openfiler is easy to install, easy to configure, and can provide both types of storage. The basic steps are to install openfiler, give it a static IP, present one disk via NFS, and present a second disk via iSCSI. While openfiler is easy to use the interface isn't intuitive for a non-storage person. Luckily the guys at labguides.com have a great openfiler lab that walks you through installation, NFS, and iSCSI.

http://www.labguides.com/storage/openfiler/

I give openfiler a small disk for the OS, then a large virtual disk for iSCSI and a large virtual disk for NFS. There isn't a reason to present multiple disk for software raid.

AD Domain Controller : This component is really optional, but I like to have a DC  available to provide AD integration for authentication in my labs. To bring up our AD DC we now get to use our Windows Server 2012 template we just created. Right click on the template, manage, and clone.

Click next to start the wizard, then select the gold snapshot we created for cloning and click next. 

I run linked clones in my workstation lab to save space. Chose the clone type and click next.

Edit the VM name to something more descriptive than "Clone of 2012 Template" and click finish to complete the clone.

When the clone is created I like to move my "Shared Infrastructure" VMs into their own folder and keep them separate from lab scenarios. Once I move the VM it's ready to power up, run through the small sysprep customization, and promote to a domain controller. It's been well over a decade since I was responsible for domain controllers, if you need further advice on this step you are barking up the wrong tree. I still keep trying to run dcpromo.exe.

Microsoft has a guide available here :

https://social.technet.microsoft.com/wiki/contents/articles/12370.step-by-step-guide-for-setting-up-a-windows-server-2012-domain-controller.aspx

At this point I have two folders, Templates and Shared Infrastructure. These folder contain the building blocks I will use to construct lab scenarios.

4. Build a Lab Scenario

When I build a lab scenario the first thing I do is create a folder for all the VMs in that lab. Down the road this is very helpful; once you run out of RAM you will want to turn one lab off to turn another lab up. The folders help keep this straight. 

I want to build a VCAP5-DCA lab, so the first thing I need are ESXi hosts. I can simply clone two ESXi hosts from the templates we created, and move them to the VCAP5-DCA lab folder. I also need a vCenter, so I will clone a 2012 server and move it to the folder as well. I prefer a windows vCenter since I will need to work with VUM. I'm now ready to add it to the domain, install vCenter, connect to my hosts, and connect my hosts to my NFS and iSCSI storage.

I use the browser and vSphere client from the host Windows 7 machine to manage vCenter. The host machine running Workstation will have access to all of the networks created.  

To add a virtual appliance like the VMA or vCenter Server Appliance click file, open, and navigate to the .ova or .ovf file and import it. 

If I want a 5.5 lab I can simply shut down my 5.0 lab, create a new folder, clone another Windows vCenter, clone some 5.5 hosts, and go to town. The shared infrastructure and templates let me spend more time with the technology, and less time getting everything set up. This framework let's me create as many scenarios as I need, adding templates as I need them.

There are even guides to get Hyper-V running on Workstation, or entire cloud frameworks like Openstack. The only limit is the amount of RAM on the box. 

Posted in: Openfiler, vCenter, VMware, VMware Nested Lab, vSphere, Workstation