Virtual Machine Monitor Execution Modes White Paper

Have you read Virtual Machine Monitor Execution Modes: in VMware vSphere 4.0?

It has some valuable information about performance of workload, VMM configuration and hardware features.  It explains how ESX decides what configuration to automagically choose, plus four cases when you might over-ride this and how.

This post extracts Cisco UCS specific observations.

ESX can use software or hardware virtualization for CPU and Memory Management, or a combination:

1. ESX software virtualization for CPU is called Binary Translation (BT), and for Memory Management Unit (MMU) is called shadow page tables (or swMMU).
2. UCS blade hardware virtualization for CPU is called Intel VT+x (or HV), and for MMU is called Intel Extended Page Tables (EPT, or hwMMU).
3. Combine software/hardware virtualization, such as hardware VT+x and software swMMU.

The Cisco UCS B200-M blade server uses the Intel 5500 series of CPUs, also known as Core i7.  You have to explicitly enable VT+x and EPT in the BIOS, otherwise it is software virtualization all the way for you (not always a bad thing, as explained here and in the VMware paper).

Just enabling these features does not mean that every VMM (i.e. every virtual machine) will use them – surprised?  It’s important to understand the following:

The VMM configuration is not just based on the underlying hardware features alone.  The automatic VMM configuration depends on two things:

1. Hardware CPU features – on UCS Blades these are Intel’s VT+x and EPT.
2. Guest OS

As an Administrator, you have a third option of manually over-riding the VMM configuration if the workload requires it.

So, assuming you have enabled VT+x and EPT in the BIOS of your UCS Blade, what is the VMM configuration that is automagically chosen by ESX for each Guest OS?  Details in the paper, but here’s a quick summary table:

Guest VMM Configuration Matrix

These “best” default configurations have been chosen based on all the performance work that VMware have done, and their customers have done, over the many years that their code has been running (and this isn’t just ESX, this is Workstation, Server etc. – they all share common code, which is why VMware stuff is bullet proof).

Note that in the VMware paper there is an order to the list of default configurations which suggests that for combinations like “FT and 64-bit guests” that the higher-order default configuration wins: in this case, FT is mentioned first and therefore a combination of VT+x and swMMU is chosen.

The third way

What about the “third way”, then?  The VMware paper explains four possible exceptions to the automagical VMM configuration:

1. Java might benefit more from swMMU.
2. 32-bit XP might benefit from BT instead of VT-x.
3. SAP in benchmarking mode might benefit from swMMU.
4. Disable VMI if hwMMU is available.

If you want to over-ride the default configuration then of course you are going to test and benchmark before and after your change, aren’t you?  To make the change is easy, you can do it via the vSphere Client or the .vmx configuration file.  Details in the paper.

Summary

VMware go to great lengths to make their products simple to understand and deploy (have you seen how easy it is do use Fault Tolerance?), but they can’t do much about the workloads you run in your organization, or the engineering and performance standards you have.

Their best bet is to help you, someone who does know your organization’s approach and capabilities, to understand more about the inner workings of ESX so that you can identify any  areas to focus on instead of having to reverse engineer everything.

VMware’s VMM Execution Modes paper achieves that by explaining what you need to worry about, and what you don’t.  Nice work, VMware.

Related posts:

1. Cisco UCS and vSphere Management: Inside, or outside?