ViewYonder

I used to work in the same team as Scott Drummonds at VMware, and I’ve always known he was in a different, much higher league than me.

This week, Scott delivered another great performance at the Burton Conference where he provides the hard data in contrast with Simon Crosby’s conjecture.  If you haven’t seen Scott present yet and your work involves virtualization, well that’s like being a blues fan and never seeing B.B. King live.

Check it out: Burton Group Analyst Conference The Thrilla in California: Debating Hypervizor Performance (hosted by the excellent Chris Wolf).

At the end of this debate, I wonder if you’ll have the same viewpoint as me?  I’d like to know what you think about the debaters, the discussion points and what you think about performance.  My takeaway is that Drummonds won because he is backed up by data; Mr Crosby on the other hand, is all conjecture.

After watching this fascinating debate, I wanted to know some more so I spent an hour doing the following:

1. Who is doing most about performance, VMware or Citrix?  Just talking about it doesn’t count.   Scott’s (valid) point is: if Citrix published their SPECweb (and all the other benchmarks) results, then customers could see and compare the two results – check mate to Scott?
2. Check out some of Simon’s previous postings, on performance, over at the Citrix website – have I got him all wrong?  Read on to find out.

Here’s what I found:

Who is doing most about performance, VMware vs. Citrix?

The easiest thing for customers to do is NOT pay good money to run their own industry benchmarks (note: this is different from running their “own workload” testing, which is vital -  a benchmark is an industry vendor-vendor comparison and does not accurately reflect each end-user organization’s unique application stacks).

So what would help customers the most is for the vendors to run industry benchmarks and submit the results to the independent industry body, such as spec.org.  These results are then analyzed by the benchmark owner and end-user customers can then easily compare vendor product A to vendor product B.  The results are not vendor biased.  So what benchmarks, published on the benchmark owner’s website, are available for ESX and Xen?  As I had no more than an hour, I though it fastest to start with what VMware were claiming and then what Citrix was claiming.

So who cares most about open industry benchmarks?  Here’s what Google told me:

According to this table, does VMware care more and has it worked harder than Citrix to help customers find objective data about their products so their customers can make the most informed decision?

If I’m looking in the wrong place to find published Xen benchmarks, then please point me in the right direction and I’ll re-run the above table.

This also got me thinking about Simon’s claim that “open source is best, because tens/hundreds/thousands/millions (pick any) of engineers are testing/fixing/maintaining (pick any).”  Does this mean Citrix doesn’t have a performance team and instead relies on the altruism of the community, and “suppliers” like IBM? There are some extremely capable Xen experts out there, but are they working like a focused, cohesive, full-time unit like VMware’s seasoned performance team?  Does this explain the lack of data from Citrix?  I don’t know the answer, do you?

In the absence of data from Simon Crosby, what is he conjecturing?

In the previous Google searches, the only results returned for citrix.com were usually posts by Simon Crosby on the community.  These weren’t posts of detailed benchmark results, in fact they were usually anti-VMware benchmark posts: but hey, isn’t that like a guy telling you how rubbish your car is, when he’s only pedaling along in his toy go-car?

If you do a Google for site:citrix.com xen performance Crosby you will find a bunch of blog posts from Simon that will give you a real understanding of where he comes from – after all, these are his words and no-one elses.  In that list of results you’ll find examples of him complaining about VMware’s benchmarks while he offers none himself, plus condescending criticism of other bloggers like Paula Rooney of ZDnet and Matt Asay of cnet.  He even picks on Brian Madden.

In the top 10 (of 412) results in that Google query you will not find anything that Simon Crosby writes that helps customers benchmark Xen against any other product, or find data to support Simon’s tired argument about “price/performance” (the sibling of “VMware’s EULA” complaint).

In the debate, I think Simon had one great point about the importance of user experience in desktop performance, and that it is hard to measure: but I didn’t get the point in the context of this debate.  Was he implying it is impossible to scientifically measure user experience, or that he understood this and Scott didn’t, or that Xen was better than ESX?

If you do a Google for site:vmware.com esx performance Drummonds you will find a bunch of blog posts from Scott that will give you a real understanding of virtualization performance, backed up by data, and some real in-depth how-to articles that help customers learn and “do” performance.  Learn more about the VMware Performance Team’s work at their blog – VROOM!

It’s standard practice for industry minnows try and get attention by throwing grenades at the market leader, in the case of Simon Crosby saying things like  “VMware’s VMmark is rubbish, ancient, useless” but the reality is that VMware has at least done something to help their customers, instead of just blogging conjecture.  I would posit that VMmark is not ancient, it is useful and it is relevant, and NEC, Unisys, HP, SUN, IBM and Dell all use it.

Time for Simon Crosby to put up or shut up: help the IT world by showing us your numbers, Mr Crosby, and use SPEC-like industry benchmarks not “Project Virtual Reality”.