Data Center Spaghetti

Data Center Spaghetti

There are quite a few different on-ramps to Unified Computing and one is particularly well known: Data Center Consolidation.

In the current economic climate, consolidation is becoming even more important than ever before, delivering substantial and tangible financial benefits that all CFOs will love.

Data center consolidation has been happening for years across the world, across business verticals, but if you could take an iPhone picture of the industry then you would find a mix of organizations that have completed consolidation as well as those who are in-progress, and many more who haven’t even started.

There are some great documents on server consolidation on VMware’s VIOPS site:

The funny (ironic, not ha-ha) thing about data center consolidation is that no matter how big your data center, you still have to do the same pattern of due diligence and planning up front – although of course with large scale, you will be doing a lot more of it!  If you have just one IT service made up of three applications running on six servers in production, and three each in staging and development (total twelve servers) then you have to do the same kind of planning work that someone else has to do even though they are looking at hundreds of services and thousands of applications and servers.  You’ll still have a three phase approach (planning, architecture, migration) with the same activities: but of course with thousands of applications, then the approach will be different: using different tools, more automation, “chunking” the migrations, and other efficiency methods required for large scale consolidation.

In Rodos’s Migration Methodology proven practice, he lists nine steps that are required whether you are migrating one or one hundred services:

  1. Initial server scan for exclusions and candidates
  2. Detailed Server Audit
  3. Planning phase
  4. Preparation of the VMware Environment
  5. Migration process development
  6. Specific case testing and POC
  7. Migrations
  8. Decommissioning
  9. Documentation and review

Around steps 2/3 in Rodos’s process, I am seeing more and more requirements for application analysis: gone are the days when running VMware’s Capacity Planner provided enough planning data.  Now virtualization is running bigger, business critical IT services and people want to consolidate those services – which means understanding how that home-grown CRM system is glued together, and what systems and services it depends upon.  This is also conversely true: before you migrate that DNS server, are you sure that the migration won’t impact dependant services?  How do you know?  Gut feel?  Instinct?  Please leave the building now, and hand your ID badge in to security: your services are no longer required.  We need data, concrete proof to base our migrations on.

The seven pieces of data analysis we need is as follows:

  1. Accurate inventory of IT assets (software and hardware), linked to their physical location, with correct owners, along with information like server lifecycle (is this server brand new or fully depreciated?).  Get someone who has a real throat to choke to sign this off as a true picture.
  2. Accurate inventory of IT services. This is what everyone means about “IT aligning to the business” – just what services does IT provide to the business, what are they called, who owns them, what are the SLAs, what’s the operational history, who’s the resident expert: understand everything, and document it.
  3. Accurate map of IT service to IT assets (applications, servers) covering all environments – production, staging, development, system test – everything everywhere.  Get the business/application owner and IT guy to jointly sign this off as a true picture.  If there is ANY doubt, then don’t go forward until it’s complete.  If you try to migrate systems that underpin business services without all the facts, just imagine yourself in six months time in the CEO’s office, on your own, in the middle of the floor, explaining why your decision that has brought the business down.
  4. Complete operational history of each IT service and IT asset.  Change records, problem records, knowledgebase items, procedures, related tools – everything.  What is “normal”?
  5. Capacity and performance history of the IT services and IT assets. Historical trends (9-5 mon fri), CPU, Mem, Disk, Net.  This is the data that VMware’s Capacity Planner is awesome at, but you need higher level application data.
  6. Interdependency map of IT assets. What servers and applications depend on the DNS server, and the logging server?  What applications are hosted on the IIS servers?  Which application servers connect to the Oracle database?  To get this data requires an analyst to work with your staff AND use a tool like EMC Ionix Application Dependency Mapping (f.k.a. SMARTS).
  7. Data center architectures including logical and physical networks. You need to know about the environment you are moving from, and where you are moving to.  For example, if this is a migration from one data center to another, then you might have to re-IP each server – what impact will this have on the applications? Will DNS be different? How do the application components talk to each other – do they use a naming system like DNS, or have the developers hard coded IP addresses?

Step 6 is where the application interdependency is required.  These applications are the strands of spaghetti that are seemingly inter-twined in a chaotic and not-documented fashion.  If you move one component (strand of spaghetti), what’s the impact on it’s neighbours / dependants?

Being a simple person, I always start by avoiding “boiling the ocean” (analysing the whole problem at once) and instead start with one IT service and go through all the stages from discovery through to migration.  By doing this successfully for one IT service, you provide the template for all migrations and get to train up the IT staff to then go ahead and do their own migrations (or delegate to a turn-key partner).  This is like sorting the spaghetti one strand at a time, but as you get better you can start to do more than one strand at a time and perform concurrent migrations in parallel.

Beware anyone who says they can migrate hundreds of servers in a few months because there are many constraints on consolidation:

  • There is a ramp-up period whilst you do the complete process from Discovery to Migration for one IT service and get the pieces working well, before you can start to do more concurrent migrations.
  • There is a pre-requisite for migrations: the target architecture must be ready and that can take many months.
  • The organization might have strict operational controls such as changes only allowed at weekends.
  • There might be a cultural resistance to consolidation which will limit the throughput no matter how efficient the process.
  • The mix of COTS (Commercial Off The Shelf) and home-grown applications will alter the complexity and therefore rate of throughput: rule of thumb is that COTS are simpler, but this isn’t always true because many COTS applications are customized in each IT shop.

Whilst all plates of spaghetti might look the same, and the general recipe is the same, there are thousands of different implementations of the spaghetti dish.   Different pasta types, different sauces, extra ingredients: this is exactly the same with data center consolidation which has a familiar pattern but the real implementation is different each time because each IT system is unique.

Related posts:

  1. The Z-Factor: Scaling deep in the data center