Before I finish work this week and get ready for the England World Cup game (yes, I’m playing in goal), I wanted to share an observation a customer made to me this week:
“So what you’re saying, Steve, is if I go with UCS then I only have one management point and one set of top of rack switches for 112 blades?”
Yes! That’s right! You can run 14 chassis, each capable of holding 8 blades each, that’s 112. Run 50 virtual wotsits (desktop, server, anything you like) per blade and that’s 5,600 virtual machines you can deploy without ever raising another purchase order for more “switches” or having to configure more “switches”.
The reason you can do this is because each chassis has logically invisible fabric extender that, as the name suggests, extends the (top of rack) fabric interconnects. These aren’t switches; they are extensions of the fabric interconnect. That means no management. Simple extension. Cloudy.
Think about it: 112 blades. 5,600 virtual machines. If you could deploy 100 virtual machines per week, that would be more than a year without any network infrastructure required. No network guy required.
Get it?
Related posts:
How much bandwidth are you talking about per chassis? 20GB?
ok, so is this w/ 6120s or 40s? Also, how many uplinks per fex? Or am I totally not asking questions that make sense?
ok, nevermind. math says you’d have to buy 6140s even if you only had 1 uplink per fex.
good post, I usually tell the client as they can focus to expand their core network and SAN, and leave the UCS to continue scale as it need.
6120 would be fine – 14 chassis = 112 blades, so that’s 14 ports on a 6120 if you use 1 uplink per fex.
that’s right – 2 x 10Gb shared by eight blades as a minimum. you could up that to 2 x 20Gb or 2 x 40Gb (and soon it will be even larger).
Great Post Steve – I have to play Devil’s Advocate for a second. Do you really see virtualized UCS solutions with only one FEX uplink? I don’t. I see 2 FEX uplinks as a minimum and 4 for heavy use.
@ 2 FEX Uplinks you need 28 Southbound ports on a 6140 and 4 isn’t possible because you would need 56 if you were consistent accross the domain.
I’m glad Cisco increased the max number of chassis but I see the 6140 being hte limiting factor now. 2 uplinks per chassis on a 6140 gets you 20 chassis (almost there) and 4 uplinks gets you 10 chassis. That is a more realistic limit.
Fair point, Aaron. You are describing the tradeoff between scalability and “performance” and the most typical config we see is two uplinks per FEX: but we have to ask “why?”. IMHO it’s an artifact of legacy thinking rather than any science, and more to do with Noah’s ark of doing things in twos… For the benefit of others (because I know you know this)…
With just one uplink per FEX you have two uplinks per chassis giving the chassis an aggregate bandwidth of 20Gbps to share amongst between 4-8 blades (because there are always two FEXs).
Increasing the FEX uplinks to two means four uplinks per chassis – a whopping 40Gbp/s! I think in most cases, and I mean 80%+, that just isn’t going to be used if you add network+storage traffic (ie. everything over ethernet).
In fact, I think the 2 x FEX uplinks is a case of over-cautious or CYA design and people who do this are robbing their customers of scalability and fantastic TCO. I really, really don’t think anyone has stated the case well enough so today I’m going to write a blog post that expands on your point about FEX uplinks as i think there are four cases (note: uniform means all chassis have same uplink count).
1. Uniform 1 Uplink – most scalable to max 14 chassis / 122 blades (constraint current supported chassis), most over subscribed with 20 Gbp/s agg. unified bandwidth per chassis
2. Uniform 2 Uplink – scalable to max 14 chassis / 122 blades with 6140, medium overscribed with 40Gbp/s agg. unified bandwidth per chassis
3. Uniform 4 Uplink – scalable to max 10 chassis / 80 blades with 6140, lowest oversubscription with 80Gbp/s agg. unified bandwidth per chassis.
4. Mixed uplink – each chassis has 1, or 2, or 4 uplinks and can be different from it’s neighbour, so the final result is anything between 80 and 122 blades.
Steve, from the interest levels here it sounds like this might be a good lab test for us, see if we can put some science behind the instincts.
what a great idea
Steve – All good points and I just read the new post, nice work again! I’m going to post a comment over there so we have a thread. Thanks!