I recently posted about how you can scale one UCS system to 122 blades, which means if you’re using it for virtual servers you could put over 5,000 on that one system and not buy any more networking hardware for a year.

There are four approaches to UCS scalability and I’ll explain each one and then moan a bit about how people are taking a Noah’s Ark approach to design, often seen applied (incorrectly) in the name of Availability (creating redundant redundancy!).

As you scale UCS you add more chassis and more blades to the Fabric Interconnects.  You don’t need to add any more network, storage or management infrastructure: JUST BLADES.

The Fabric Interconnects at the top of rack devices are the brains that do switching and so much more.  You connect cables from these Fabric Interconnects to ports on Fabric Extenders (FEX) that sit in each chassis.  Think of a FEX as a logical extension of the Fabric Interconnect: a FEX is not a switch, it doesn’t need individual management, it’s an extension of the fabric.

Everytime you add a chassis, then, you are filling up ports on the Fabric Interconnects and you have a finite amount of these – either 20 (if you have a 6120) or 40 (if you have a 6140).  There are also current limits (tested supportability limits that Cisco will guarantee for you) of 14 chassis per UCS (this gets bigger with each release).  These are the two scalability constraints: ports and supported chassis, leading to four scenarios:

1. Maximum scalability/oversubscription with a 6120 and 1 uplink per FEX gives 14 chassis / 122 blades and 20Gbp/s aggregate bandwidth per chassis / 8 blades.
2. Medium scalability/oversubscription with a 6140 and 2 uplinks per FEX gives 14 chassis / 122 blades and 40Gbp/s aggregate bandwidth per chassis / 8 blades.
3. Minimum scalability/oversubscription with a 6140 and 4 uplinks per FEX gives 10 chassis / 80 blades and 80Gbp/s aggregate bandwidth per chassis / 8 blades.
4. Mixed scalability/oversubscription with a 6120 or 6140 with each chassis having a choice of 1, 2 or 4 uplinks from the FEX (e.g. chassis 1 has 1 uplink, chassis 2 has 2 uplinks).

Or in a picture:

Important note: at all times there are redundant uplinks from each chassis.  So when I say “1 FEX uplink” I mean _PER FABRIC_ and there are two fabrics, A and B.  Each chassis has two FEXs (one per fabric) that connects to their “own” 6120/6140 fabric interconnect.

A common problem with architecture is that people “double count for redundancy” or create “redundant redundancy” by deploying redundant components (e.g. RAID arrays) in already redundant containers (e.g. Web server that is one in a redundant farm).

The same thing tends to happen in UCS because people think that “two FEX uplinks is more redundant than one” which isn’t true at all.   Redundancy comes from two FEX devices per chassis, not two links per FEX.  More FEX ports means lower oversubscription (at a cost).  That’s all it does.  Think:

• Redundancy goes left-to-right (across FEXs), and;
• Oversubscription goes top-to-bottom (across FEX ports).

So, now we’re all over the confusion between redundancy and oversubscription, what’s the cost/benefit of these four scenarios as seen in the above diagram?

• Maximum scalability (122 blades) has a lower TCO because you are sharing one UCS between 14 chassis which means you shouldn’t have to buy another UCS until you’ve deployed over five thousand VMs.  It also means less cable costs (SFP+ from FEX to Fabric Interconnects).  You will need to buy and extra 12 ports licenses for the fabric interconnect (8 are bundled with 6120).
• Medium scalability (which should only be deployed if you need the aggregate bandwidth of 40Gbp/s per chassis!) means you can get 10 chassis out of one 6120, but you’ll need a 6140 to run 14 chassis because that’s 14 x 2 = 28 ports.
• Low scalability means you can only get 5 chassis out of a 6120, and only 10 with a 6140.  This uses all ports, using the maximum amount of cables, and you need extra port licenses to use all the ports on the fabric interconnects.  Most expensive option.
• Mixed scalability means you can deploy chassis with different oversubscription and this drives a different TCO.  If you had three chassis with Mixed Scalability, then Chassis 1 might be Maximum Oversubscribed (with 1 FEX link per fabric), Chassis 2 might be Medium (2 FEX links) and Chassis 3 might be Lowest (with 4 FEX links).

Money is finite and so are network ports.  Your job as a systems designer/architect is to make sure the system has adequate redundancy and oversubscription and in most cases that will only require one uplink per chassis per fabric.  The fabrics give you redundancy, and the FEX ports give you bandwidth.

Many people are deploying two FEX ports because they don’t know or can’t predict the profile of their unified traffic (ethernet and FC-over-Ethernet) and the middle ground feels safer (no science).

You can always change your mind later anyway, so you can ramp up (say from 1 uplink to 2 or 4) or ramp down (say from 2 uplinks to 1) should your data show that your workload has a bigger/smaller profile.

And perhaps that is the best point of all of this:  by all means work out the tradeoff between scalability and oversubscription but remember, most importantly of all, it’s really simple to change it later (add/remove cables and re-acknowledge the chassis – DONE!).

Related posts:

1. With UCS you buy one set of switches every 112 blades
2. How real life became my worst nightmare