Hello, I'm the FNG

Hello, I'm the FNG

I’m still The Effin New Guy (The FNG – something I learned from my old pal Lane Inman at VMware) at Cisco, and I want to share some of my early lessons about UCS is, and what it is not.  This first lesson is about Why a Cisco UCS 6100XP is not a Switch.

Remember how everyone used to (some still do, and they are being hunted right now!) think that ESX is “based on Linux” therefore “ESX is Linux”?  If you have never heard it, or worst of all think it, then let me remind you how this myth arose.  First, the COS (the admin bit that used to boot the server) is a cut down Red Hat OS.  Second, the drivers in the vmkernel have Linux history and I think vSphere drivers are related to Linux 2.6.  Does that make ESX into a general purpose Linux OS?  No.  Same deal with the 6100XP:

Just because the UCS 6100XP Fabric Interconnect has some NXOS code in it does not make it a switch: the 6100XP does many more things than “just” L2 switching.

This is evolution, baby.

When I say 6100 I mean the UCS Fabric Interconnect which can be a 20-port 6120XP or a 40-port 6140XP. There are two modes that a 6100 can run in.

End-Host Mode

Think of your UCS (1-40 chassis) as a dynamically changing (bigger/smaller) SINGLE host – all those chassis are one chassis – with LOTS of network inferfaces presented to the outside world.

How many is LOTS? Think of 8 chassis x 8 blades x 1 Palo NICs x 50 vNICs = 3200

Note:  It’s a bit more complicated than that, and I picked 8 chassis from thin air, but a good rule of thumb.

Think of the Fabric Interconnects as a big network bus stuck on the back of your commodity mainframe.

Any network related changes (e.g. adding a VLAN to a blade NIC) are handled AUTOMAGICALLY and nobody has to configure anything else.  The upstream Fabric Interconnects are automatically configured to make sure that blade can get to/from the VLAN the Service Profile guy configured.  Easy peasy, lemon squeezy.

Although in End Host Mode you can SSH to the Fabric Interconnects and connect to an NXOS session, don’t think you can do anything other than “get” values.  Just like ESX Service Console is a cut-down version of RedHat, the NXOS in a 6100XP is not a full NXOS.

In End-Host Mode, the 6100′s only learn the MAC addresses of the devices within itself – it doesn’t learn the MAC addresses of anything “upstream” in the outside, northbound LAN.

Any uplinks / border ports you configure (to the upstream LAN) are automatically configured as .1q trunks.  No action needed by any administrators.

The 6100s do some switching – they can switch between servers in the same 6100, but if the servers are using different 6100s (server A’s active NIC might be going via FI-A, and server B via FI-B) then the switching has to be done outside of UCS.

EHM represents two significant benefits of UCS:

  1. EHM massively reduces administrative overhead, which as well as reducing the amount of boring changes it minimizes the chances of human error, freeing up said humans to do more interesting, creative things.
  2. EHM allows the system to scale massively because you aren’t clogging up the control plane with unnecessary layer 2 switching dynamics (like STP).

Switch Mode (I call it Pseudo-Switch Mode)

You can do a few more things with the Fabric Interconnects when they are in Switch Mode so if you’re a control freak, and most of us are, you will be sorely tempted to use this mode.  Resist! Resist!  Or I shall cut off thy hands! ;-)

The 6100s can now do more L2 networky stuff so you have the potential to break do more – but you still have little control over the default behaviour (less to break, it works out of the box etc).

The upstream MACs are now learned which means STP is now switched on to prevent those pesky loops, but don’t think you can logon and start issuing those NXOS configuration commands: you can’t change Bridge Priority, Hello Timers…

An additional feature available in Switch Mode is the ability to attach some things directly into the UCS, such as Network Attached Storage.  You still need an upstream, external fibre channel switch to get to your FC arrays though.  When we say L2 switch we are talking about LAN switching.  Remember that FCoE currently only happens within UCS – when network traffic leaves UCS (via the 6100s) it is either LAN (L2) or SAN (FC) traffic (more on that in another post later p’raps).

OK, so it’s not a switch, but which mode do I choose, and why?

In most cases, use the default End-Host Mode and move on to something more interesting, creative and valuable.

You only need Switch Mode if the following is true:

  • You want to connect a NAS directly to the 6100s.. Noooooooooooooooo….
  • You have a disjoint LAN above UCS because EHM will not expect nor handle such craziness… ;-)   Your outbound traffic is dynamically pinned which means EHM expects the same L2 network consistently across all the uplink ports.
  • You connect the 6100s into a pair of HSRP routers that are STP roots on different VLANs.  Normally, the 6100s connect into your garden variety Catalyst or Nexus switches though…

There you have it. Just like ESX aint Linux, Fabric Interconnects aint Switches.