Deployment and Change Management - The Problem

I had been hoping to get together with people at Drupalcon this year to discuss issues surrounding deployment and change management. Unfortunately I will not be able to go. So this will have to act as my contribution to that discussion, which will hopefully carry on well beyond Boston. There are three parts - The Problem, Things That Help Now, and Some Ideas For The Future.

One of the first things we tackled while putting together our Drupal architecture was workflow - both for our editorial team and our engineering team. Very quickly we knew we had a problem on our hands.

Our development environment is pretty typical for any large site. We have three server environments - dev, qa, and production. For us, dev is one box where every developer has a sandbox which is basically a VHost and an svn working directory. The code there connects to a local mysql database. Every developer has root on this MySQL server so we can create, trash, and share databases at will. When code is ready for testing we have a packaging script that pulls it together and pushes it to the qa server, where it is tested. This process is repeated until code is ready to go to production when it is packaged and pushed live to a series of load-balanced production web servers.

Editorially we had around 10 separate content management systems for various parts of the site that had been built up as needed over the years. This included things like Movable Type and phpBB as well as custom solutions. The editorial team passed Word documents around through the various phases of writing and editing, finally saving them as text and pasting them into whatever system was in place for final deployment. The plan was to move all this stuff into Drupal, taking advantage of revisioning and workflow.

The problem comes when you want to deploy new functionality. If there was one thing that was nice about the old way, it is that everything was isolated. Recently we upgraded our user registration system which involved some major database munging. This was not a huge deal, we just took registration and login offline for a few hours overnight (logged in users remained logged in) and did our thing. The rest of the site went on its merry way. Obviously this doesn't fly in the new world.

A great deal of Drupal's life is stored in the database. This is even more true when you're extensively using CCK, Views, and Taxonomy (and we are.) Simply deploying code is not enough to push new functionality live anymore, you need to push the database too. However you can't just replace the database every time you want to deploy new stuff - even if you refreshed QA a day or two or even a few hours before launch, you risk stomping on any new comments, logs, cache, users, etc. Drupal is constantly generating new content.

Also how does the editorial team stage their content now? They can't stage it on QA, we need to be able to have pretty much free reign on that database while testing and debugging. Even if they did how would the content move live? The problem is extremely sticky and we struggled with it for a long time before coming up with a solution nobody really liked but which seems pretty standard in Drupal-land.

The process for deploying code is unchanged, and works well. However where data is concerned the process is the opposite - Editorial does all their work on the live site. When necessary we refresh dev and qa with this database. In a nutshell, code moves "forward" and data moves "backwards." In a situation where code relies on, for instance, a new content type or view or something, code is pushed forward and the changes are made by hand through the admin.

This works but it is unweildy, and the more changes you're doing by hand the worse it is. We are working on a project now which will involve deploying 9 content types, perhaps twice as many views, a new taxonomy vocabulary, a full new set of pathauto settings (since we're upgrading to 2.0), and a wide variety of assorted admin changes in different areas. It also doesn't account for the fact that if something goes wrong while you're doing your final deployment, there is no easy way to roll back to square one. Hearing stuff like this makes business owners twitchy.

So what can be done? Stay tuned.