ViewYonder

An old mate from my web hosting days has “CAT5″ tattooed on his arm and a head shaped like an RJ-45. Ok, the last bit was made up, but you get the point. He lives and breathes networking, and much like physicists look down upon every other branch of science (but aren’t the mathematicians king?) he has little regard for other branches of IT – especially server administrators, who he often refers to a “Gollums”.

You can imagine the distaste in his mouth when people without Cisco qualifications tell him how he needs to change his treasured network design and policies, and actually give up some of his responsibilities to the server admins – gulp! But that’s what I’ve been helping customers do over the past five years, and you might think that in five years we’d have progressed past this barrier, but we haven’t and only now the light is at the end of the tunnel with the Cisco Nexus 1000V – more on that later, but first, what is all the trouble about? Why not just go with the flow and not change the network implementation?

If you DON’T use trunk ports and VLANs between ESX and the Switch, then you have to either increase physical NICs (one per network) in the ESX host, OR reduce the number of virtual machines in different L2 networks). That has a direct impact on TCO (more NICs/ports) and ROI (lower guest:host ratio).

So let’s look at some key points in the VMware-Network discussion:

1. No trunk ports at the access layer, and other “antiquated” policies
2. Understanding vSwitches and ESX.
3. STP or no STP, that is the question.
4. How much bandwidth do you need?
5. How can life be better?
6. Cisco Nexus 1000V

If you are (a) a network engineer, or (b) a server admin/VMware person, then I highly recommend you read Ken Cline’s fantastic blog series, The Great vSwitch Debate. Ken likes to debate with himself, which is a Good Thing, and he’s managed to succinctly and correctly get over all the major discussion points on VMware networking.

No trunk ports at the access layer

This is one of the examples of an existing design / policy directive that is impacted by the paradigm-shift that is server virtualization. If you don’t know what I’m talking about, then you need to read Ken’s vSwitch Debate – but in effect, the Network Engineer needs to configure his network switch differently for ESX than for all the other server types. If that’s against policy, why should it be done? (arms crossed, chin set).

One way of overcoming this barrier is to imagine that an ESX server is like a blade chassis, and that the NIC of a vSwitch is like a mezzanine uplink port, and therefore lots of servers (doesn’t matter if physical or virtual) will be sending/receiving data via this uplink port – ie. many MAC addresses will be using it – then the network guy usually gets it: to get the ROI from virtualization, trunk ports are essential, and they work – do whatever testing is required, but design for trunk ports.

Trunk ports are good way to then get into the detail of how ESX and vSwitches work – remember Ken’s vSwitch Debate. But we can cover some quick points here.

VLANs come under this banner too – with ESX you can do 802.1Q (VLANs) in the guest, the host or in the external switch. The most common way with ESX is in the host (at the ESX level) by configuring port groups in a vSwitch, and giving them an ID the same as the VLAN number. This way, you connect VMs to Port Groups and they are automatically 802.1Q tagged as being in a specific VLAN – easy! The Network Engineer just needs to share the VLAN IDs, and configure the external switch port to allow a range of VLANs on the port. Easy!

Understanding vSwitches and ESX

This blog post is not an education on VMware networking (I’m having to bite my tongue and try to keep an eye on the prize).. With regards the Network Engineer barrier, some key things to be understood between a VCP and a CCNA – any of these, if not agreed, make up the barrier:

1. A vSwitch is a very simple software device that is managed via vCenter or the COS command line, BY THE SERVER ADMINISTRATOR. No IOS in ESX3.x.
2. THE SERVER ADMINISTRATOR connects a Virtual Machine’s virtual NIC to a virtual switch when the virtual machine is created in vCenter.
3. THE SERVER ADMINISTRATOR can reconfigure the virtual NIC of a Virtual Machine through vCenter at any time that the Virtual Machine is up and running.
4. The Virtual Machine has a unique, dynamically generated MAC address that might change if the Virtual Machine is moved.
5. The MAC addresses on layer 2 datagrams seen on the external switch port will be the MAC addresses of the virtual machines for their traffic, so think of the ESX physical NIC as acting like a bridge.
6. When a VM is power on it is assigned to a physical NIC for the rest of the time that the VM is powered on that host, unless using IP-Hash balancing, and unless that physical NIC fails.

For more information, read Ken’s vSwitch Debate – it rocks!

STP or no STP, that is the question

Spanning Tree Protocol (STP) is a network implementation that prevents layer 2 loops by collapsing redundant paths into a tree. Imagine two switches, that are connected together with an Inter-Switch Link (ISL) in an active-active “cluster”. They expect to see source MAC addresses (ie. the “sending” server) from only one port in their configuration. If they see duplicate MAC addresses then the collapsed tree must be wrong so STP (rather, the switches in the network) need to re-calculate the collapsed tree from all the redundant paths – AND STOP NETWORK TRAFFIC WHILST THEY DO SO!

A common question and misconception with vSwitches is that, because you can attach two physical NICs to a vSwitch (these are the bridging “uplinks”, out of ESX and to the external switches), and those physical NICs can connect to their own external switches (and those switches are connected together using an Inter-Switch Link) then a virtual machine sending data via that vSwitch might flood or flip-flop MACs via both pNICs and cause a STP loop, because both switches can see the same source MAC address at the same time.

The quick answer (also covered in Ken’s vSwitch Debate) is to understand that while a vSwitch will accept any MAC addresses on any physical NICs (inbound), it will only send a VM’s MAC address out of one physical NIC (ie. will not “flood” the MAC out of all NICs). The exception to this rule is using IP-Hash load balancing with ether channels. We’ve never had STP loops with vSwitches, the software can’t do it, and the ESX3.x vSwitch is dumb when it comes to negotiations – it can’t “talk” STP. No respectable Network Engineer has to take this on face value, and they are absolutely within their rights to test and prove it in their environment.

vMotion, DRS and HA makes this more interesting – they put a balaclava on your VM and it turns all Ninja, being able to leap from servers to server. In effect, you can “confuse the tree” when you migrate a VM from one host to another and now present a MAC address at two different switch ports in the same layer 2 domain. Remember the last thing that’s done at the end of a vMotion is a “promiscuous ARP” which tells the external switch, on behalf of the VM, “Hey, I’m over here now!” If STP is enabled, then the tree recalculation will have to happen and network performance dies. If you configure something called “portfast” on your switches, and disable STP on that port, then you will avoid this situation.

How much bandwidth do you need?

If you are asking a Network Engineer for GigE or 10GigE ports, then they will be curious (tongue in cheek) to understand why you need all that bandwidth. Admittedly, an oft-ignored statistic of server consolidation is the massive under-utilization of network ports – even worse than the ~10% CPU utilization, the network is usually below 5%.

This is where data is key, and if you have been running a Capacity Planner project then you should have all the data you need with regards to the consolidated physical servers network requirements.

But you will still have a guestimate number for as-yet-undeployed and net-new servers, and you can only do your best for those – some suggestions:

• Ask the network engineer for data – if you plan on deploying web, app and DB servers, then pick some existing physical servers and get the port / usage data from those.
• If you are already hosting web, app and DB then use the stats from VC.
• Be sure to look at peak loads, not averages.
• Ask about how the network engineer might know ways to expand or reduce capacity using QoS to provide flexibility – he/she might be able to give you 10GigE links but limit you to 2GE so you can’t overload the backplane.

How can life be better?

From a Network Engineer’s perspective, has life just got a whole lot worse? Is it worth it, for the Network Engineer, to change their current practices to adapt to virtualization? Like anything in life, if this barrier is treated objectively, without religion (CCNAs are better than VCPs) or emotion (I designed this network, it stays like this), then the best way is to simply apply common sense – ie. what is the industry standard? what is the cost-benefit of the various options?

If you spend some hours doing a joint VMware-Network workshop – perhaps something like the VI3.Blueprint Networking Workshop. The life with VMware should be better for a networking engineer because:

1. You have less server ports to manage, less itty-bitty and mundane requests for servers ports – just allocated a handful of uplink/trunk ports to ESX and now the server team does the individual server allocations.  The network engineer sets the new policy, implements it with the VCP in the network + vSphere.
2. You can manage QoS much better because instead of thousands of hardly-used access ports, you have fewer “aggregate uplinks” from ESX.
3. Consolidating old servers means removing the obsolete switches and racks they live in – time to tidy up that datacenter!
4. Hardware refresh – it is common for ESX to connect to next Cisco 65xx series switches, so new toys to play with
5. The ESX3.x vSwitches are being replaced with Network Engineer-friendly Nexus 1000V.

The last point is the killer: now, with Nexus 1000V, the Network Engineer can take back control of the network and use familiar tools and processes with VMware solutions – perfick!

Cisco Nexus 1000V

If you’ve been living in a cave in Afghanistan you might not be aware of the latest vSphere product release from VMware, or the Cisco tie-up with the Nexus 1000V (NXKV for short).

The NXKV kills the Network Engineer barrier dead, gone, and hopefully soon forgotten.

Show the Network Engineer the following material, and all will be good in the world! Thanks to VMware and Cisco for killing this barrier – why didn’t you do it five years ago and save me and all my customers these discussions, hahahaha!

• Virtual Networking Features of the VMware vNetwork Distributed Switch and Cisco Nexus 1000V Series Switches
• More information on the VMware vNetwork Distributed Switch
• More information on the Cisco Nexus 1000V Series Virtual Switch
• Technical Resources for Virtual Networking