Provoking IT from Good to Great
Cisco 6100XP aint Switches like VMware ESX aint Linux
Hello, I'm the FNG
I’m still The Effin New Guy (The FNG – something I learned from my old pal Lane Inman at VMware) at Cisco, and I want to share some of my early lessons about UCS is, and what it is not. This first lesson is about Why a Cisco UCS 6100XP is not a Switch.
Remember how everyone used to (some still do, and they are being hunted right now!) think that ESX is “based on Linux” therefore “ESX is Linux”? If you have never heard it, or worst of all think it, then let me remind you how this myth arose. First, the COS (the admin bit that used to boot the server) is a cut down Red Hat OS. Second, the drivers in the vmkernel have Linux history and I think vSphere drivers are related to Linux 2.6. Does that make ESX into a general purpose Linux OS? No. Same deal with the 6100XP:
Just because the UCS 6100XP Fabric Interconnect has some NXOS code in it does not make it a switch: the 6100XP does many more things than “just” L2 switching.
This is evolution, baby.
When I say 6100 I mean the UCS Fabric Interconnect which can be a 20-port 6120XP or a 40-port 6140XP. There are two modes that a 6100 can run in.
End-Host Mode
Think of your UCS (1-40 chassis) as a dynamically changing (bigger/smaller) SINGLE host – all those chassis are one chassis – with LOTS of network inferfaces presented to the outside world.
How many is LOTS? Think of 8 chassis x 8 blades x 1 Palo NICs x 50 vNICs = 3200
Note: It’s a bit more complicated than that, and I picked 8 chassis from thin air, but a good rule of thumb.
Think of the Fabric Interconnects as a big network bus stuck on the back of your commodity mainframe.
Any network related changes (e.g. adding a VLAN to a blade NIC) are handled AUTOMAGICALLY and nobody has to configure anything else. The upstream Fabric Interconnects are automatically configured to make sure that blade can get to/from the VLAN the Service Profile guy configured. Easy peasy, lemon squeezy.
Although in End Host Mode you can SSH to the Fabric Interconnects and connect to an NXOS session, don’t think you can do anything other than “get” values. Just like ESX Service Console is a cut-down version of RedHat, the NXOS in a 6100XP is not a full NXOS.
In End-Host Mode, the 6100′s only learn the MAC addresses of the devices within itself – it doesn’t learn the MAC addresses of anything “upstream” in the outside, northbound LAN.
Any uplinks / border ports you configure (to the upstream LAN) are automatically configured as .1q trunks. No action needed by any administrators.
The 6100s do some switching – they can switch between servers in the same 6100, but if the servers are using different 6100s (server A’s active NIC might be going via FI-A, and server B via FI-B) then the switching has to be done outside of UCS.
EHM represents two significant benefits of UCS:
- EHM massively reduces administrative overhead, which as well as reducing the amount of boring changes it minimizes the chances of human error, freeing up said humans to do more interesting, creative things.
- EHM allows the system to scale massively because you aren’t clogging up the control plane with unnecessary layer 2 switching dynamics (like STP).
Switch Mode (I call it Pseudo-Switch Mode)
You can do a few more things with the Fabric Interconnects when they are in Switch Mode so if you’re a control freak, and most of us are, you will be sorely tempted to use this mode. Resist! Resist! Or I shall cut off thy hands! 
The 6100s can now do more L2 networky stuff so you have the potential to break do more – but you still have little control over the default behaviour (less to break, it works out of the box etc).
The upstream MACs are now learned which means STP is now switched on to prevent those pesky loops, but don’t think you can logon and start issuing those NXOS configuration commands: you can’t change Bridge Priority, Hello Timers…
An additional feature available in Switch Mode is the ability to attach some things directly into the UCS, such as Network Attached Storage. You still need an upstream, external fibre channel switch to get to your FC arrays though. When we say L2 switch we are talking about LAN switching. Remember that FCoE currently only happens within UCS – when network traffic leaves UCS (via the 6100s) it is either LAN (L2) or SAN (FC) traffic (more on that in another post later p’raps).
OK, so it’s not a switch, but which mode do I choose, and why?
In most cases, use the default End-Host Mode and move on to something more interesting, creative and valuable.
You only need Switch Mode if the following is true:
- You want to connect a NAS directly to the 6100s.. Noooooooooooooooo….
- You have a disjoint LAN above UCS because EHM will not expect nor handle such craziness… Your outbound traffic is dynamically pinned which means EHM expects the same L2 network consistently across all the uplink ports.
- You connect the 6100s into a pair of HSRP routers that are STP roots on different VLANs. Normally, the 6100s connect into your garden variety Catalyst or Nexus switches though…
There you have it. Just like ESX aint Linux, Fabric Interconnects aint Switches.
Related posts:
| Print article | This entry was posted by Steve Chambers on 28 September, 2009 at 23:35, and is filed under UCS. Follow any responses to this post through RSS 2.0. You can leave a response or trackback from your own site. |
about 11 months ago
Just a quick note on the “disjointed LAN above UCS” reasoning for using switch mode. You can actually deal with that quite easily in EHV mode, by using pin-groups. Place all of your “LAN A” uplinks in one pin-group, and all of your “LAN B” uplinks in another. Simply select the appropriate pin-group when defining your blade’s network connectivity, and your blade will only auto-select amongst the appropriate uplinks.
So as you astutely pointed out, resist the urge at all costs to use Ethernet switch mode.
about 11 months ago
Great point, Dave. It’s “doable” but has an admin overhead, bit more complexity etc.
about 4 months ago
@Dave Alexander
Actually that won’t work for ESX. If you use pin groups on a disjointed l2 environment the VMs will be lost on a fabric failure. The host mac address will get moved but nothing is in place to do the GARP for the VMs to move their MAC addresses. This issue is the 6120 randomly selects an uplink to be the designated receiver so you will be lost during a failure for some ammount of time (chatty VMs will fail quickly but other will be lost until they ARP). This may not be a big deal for some. It will probably function better in switch mode but then you introduce l2 re-convergence for the vlans you are bringing into the UCS.
The fix coming is to allow you to select the designated receiver port for each VLAN individually. Until that is in place best practice is to have all your uplinks carrying the same VLANs.