Install and run a full functional VMware ESX Hypervisor within a KVM virtual machine

This post is about how to setup and configure a VMware ESX Hypervisor within a KVM virtual machine. The resulting ESX Host is going to be fully functional allowing to create (nested) virtual machines with a working network connection. 

Some (funny) facts:

• Running VMware ESX within a KVM virtual machine allow memory over-commitment. That means even if your KVM Host system only has 2 GB memory you can assign e.g. 4 GB memory to the ESX Host VM.
• KVM provides the hardware requirements for VMware ESX even if ESX does not supports the KVM Hosts hardware e.g. it allows to install and run a VMware ESX system on almost any kind of Desktop hardware which is normally not supported by ESX e.g. because of an unsupported network card model.

 Steps to setup VMware ESX within a KVM VM

• Install Linux on a physical system (I used latest Ubuntu 14.04)
• Install the qemu/kvm package 

                            apt-get install qemu-kvm

•  Edit /etc/modprobe.d/kvm-intel (or kvm-amd for AMD cpus) and add

                            options kvm ignore_msrs=1
                            options kvm-intel nested=y ept=y

•  Download qemu-2.0.0 from http://wiki.qemu.org/Download and apply the following patch to hw/i386/pc_piix4.c line 206. Change

   pc_basic_device_init(isa_bus, gsi, &rtc_state, &floppy, xen_enabled(),
      0x4);
          to
   pc_basic_device_init(isa_bus, gsi, &rtc_state, &floppy, TRUE,
       0x4);

• Configure the qemu 2.0.0 source with the following command:

./configure --enable-kvm --target-list=x86_64-linux-user,x86_64-softmmu

• Create /usr/local/etc/qemu/bridge.conf

                     allow all

• Install the bridge-utils package

                    apt-get install bridge-utils

• Configure a network bridge in /etc/network/interfaces as following:

auto br0
iface br0 inet static
address 192.168.88.135
netmask 255.255.255.0
network 192.168.88.0
broadcast 192.168.88.255
gateway 192.168.88.1
dns-nameservers 192.168.88.1
bridge_ports eth0
bridge_fd 0
bridge_hello 2
bridge_maxage 12
bridge_stp off
 

 Please notice that your ip-configuration may be different

• Choose a directory for the KVM virtual machine and create the virtual machine harddisk with the following qemu-img command:

                     qemu-img create -f raw esxi00.img 16G

• Download VMware ESX Hypervisor 5.5 from the VMware website and put the install iso file (VMware-VMvisor-Installer-5.5.0.update01-1623387.x86_64.iso) into the same directory as the harddisk file was created a step before

• Start the KVM virtual machine with:

/usr/local/bin/qemu-system-x86_64 -enable-kvm -cpu host -m 4096 -smp 2 -cdrom VMware-VMvisor-Installer-5.5.0.update01-1623387.x86_64.iso -netdev tap,helper=/usr/local/libexec/qemu-bridge-helper,id=hostnet0 -device vmxnet3,netdev=hostnet0,id=net0 -hda esxi00.img

Please notice the "vmxnet3" virtual network device type!

• Go through the (normal) VMware ESX installation procedure
• After the installation enable the SSH service on the ESX Host with is now running with the KVM VM. Then ssh into the ESX Host and set the following two parameters in /etc/vmware/config:

                      vhv.allow = "TRUE"
                      hv.assumeEnabled="TRUE"

Now please reboot the ESX Host system to activate the new configuration!

• When creating a virtual machine on the (nested virtualized) ESX Host please make sure to add the following two parameters to the VM's vmx configuration file:

                       vhv.enable = "TRUE"
                       vmx.allowNested = "TRUE"

Congratulations! 

You have successfully virtualized a VMware ESX Hypervisor with KVM!

I hope you now enjoy your fully functional VMware ESX Host

Additional informations on the VMware parameters used in this howto

• vhv.allow = "TRUE" does nothing on ESX 5.1 and later
• vhv.enable = "TRUE" is only going to be useful if you want to run VMs under the VMs running under ESXi running under kvm
• You can add vmx.allowNested = "TRUE" to /etc/vmware/config in the ESXi VM to avoid having to put it in every nested VM's configuration file.

This quick howto was created with informations from https://communities.vmware.com/thread/451412
Special thanks to jmattson for the additional details on the VMware parameters