ViewYonder

VCP caught by the CCIEs

If there’s one thing that differentiates virtualization from other technologies, such as management products or applications, it’s that it’s easy to measure (and predict) that the more you use it the more value you derive, like avoiding capital expenditure: there is a direct correlation between virtual machines deployed and value derived.

Therefore, the goal of all virtualization projects should be to virtualize as much as possible: not just to line the pockets of VMware account managers, but to line your own pockets when you can rip out those old racks, and even whole data centers, and instead of deploying 500 physical units you can deploy 50 or 25, or even less than 10 now with the latest vSphere.

But, have you ever tried this?  I have.  Many times, in different countries with different organizations.  And the same thing keeps happening: the deeper virtualization penetrates the data center, the more subject matter experts (SME) one meets.  These are not just techies, like VCPs, CCIEs and the rest, but process/management like ITIL and Prince2 people.

The challenge in rolling out virtualization is that you can’t do it in isolation:  all of these SME roles are impacted in some way, either in the justification, design, implementation or operational phases.  At a minimum, the processes they use will be changed, and that spells trouble.  Rarely in my five years at VMware did I find a cohesive system of processes that I could tweak because they were all at VMM level 5.  Instead, I continuously found subject matter experts hoarding the knowledge in their brains, acting on “experience” (read: gut feel), and not having the time to document/improve processes.  Often the biggest barrier to a virtualization project was the VMware Fanboy, because that individual was awesome at what they did and didn’t see why anyone else (like Operations, or Security) needed to know about virtualization… that was Somebody Else’s Problem.  Sigh.

One day I had a moment of epiphany when on a conference call with a Japanese organization.  We were discussing the process changes and the people changes to make large scale virtualization successful, and described the VMware Center of Excellence where roles are defined and their responsibilities and training identified.  The Japanese organization had a great point:

We don’t want to invest training into one individual; instead, we want to invest training into the system.  The people improve the system, there are no role specialities or hoarding of knowledge like certified individuals.

This struck me at first as being counter-intuitive, especially to Europeans and North Americans: certifications are strongly desired by professionals, and clearly expressed as a requirement in job advertisements: but hey, have you noticed how some employers don’t like investing in training and certifications?

Vendors love certifications because it increases the success of their technology because it is implemented correctly, but also because that individual who is certified is now emotionally/religiously attached to that vendor (possibly).

What’s happening here is that SMEs are professional IT gladiators.  A VCP may use different weaponry than a CCIE or an ITIL Master, but they still get rewarded for their skills and abilities: not necessarily their improvement of the IT system which they work in.

I think the following observations are typical of many large enterprises, and probably small/medium enterprises too:

1. A professional’s value is directly tied to their individual certifications and knowledge, not their demonstrable impact in improving an IT system from good to great.  Does this help organizations?
2. Organizations (read: managers) like Gladiators: they don’t want to train them, but they will buy in talent and burn it out, rewarding glorious battles where much hair is burned running around a data center.  Are organizations hurting themselves?
3. Gladiators cannot create a system of processes whilst they are rewarded for heroics and not predictable and repeatable (read: mature) systems.  Documentation and process is low priority, or Somebody Else’s Problem.  Gladiators pick their own battles.
4. Tools are like gladiator weaponry: we all know that tools are not the answer, but yet organizations spend millions on them all the time, often on competing kit that often sits on the shelf gather dust unless an SME picks it up and runs with it.  The cost and variety of tools is inversely proportional to the improvement in the IT system.  How many CMDBs do you have?
5. Gladiators like to fight each other.  My hypervisor is smaller than yours; my security scanner can penetrate better than yours.  Often carried out under the disguise of “Product comparison / evaluation”: this is simply back-street, illegal, bare-knuckle fighting that takes value out of the IT system.

I’m about to start a new role with Cisco and help customers transform their data centers, and the systems and services that run on top of them, and I know I’m going to meet lots of gladiators, and I know that the success of projects is at risk because of them.

For example, gladiatoral VCPs can slow the progress of virtualization: how can that be?

• By not applying virtualization in places it is needed because they don’t want their baby to go on the Error Report – like avoiding Tier 1 applications.
• By not sharing their knowledge: one or two people cannot run a decent (500+ VMs) virtual data center, but how much time do they spend enabling others / the IT system?
• By being a one-trick-pony, full-time virtualization expert, little or no time is spent learning other skills and improving the IT system because they are Somebody Else’s Problem: like capacity management, or SLAs, or Operations.

And if virtualization is slowed, it costs money in the form of opportunity cost: if you have spent $5m on your virtual infrastructure to build three clusters of 16 hosts, then your break even might be around 11 VMs per host, or 528 VMs in total – or it might be twice that number, depending on your IT system’s economics.  What happens if you can’t hit that number?  What happens if the CIO demands twice that number?  You can only blame each other for so long…

I think the change has to come from the organization, from the top, from the senior management.  Instead of rewarding gladiators, reward improvements to a predictable, repeatable IT system that increases throughput (more projects, more change, more availability) at a lower cost (capital and operational expenditure).  But change is also required in the approach of the IT staff, who enjoy developing their own world-class IT system and not just their own list of certifications.

Perhaps if certifications were tied to an IT system (not a single technology, but the organization’s IT system – like ISO20000, for example) through than an individual, then the gladiators might be freed from their daily battles, and Harry S. Truman’s quote (mentioned in Jim Collins’ Good to Great), might come true in data centers around the world:

You can accomplish anything in life, provided that you do not mind who gets the credit.