Deployment And Change Management - A Framework

A couple days after I posted my last article, I went to del.icio.us to see if anyone had bookmarked it. A user named timbaileylondon had, with this note - "We need to get on top of this. The last one in particular made me cringe." Too true on both counts.

A few weeks ago I was trying to chase down a bug in content copy, because it was causing us problems with some content types we were trying to move. I had never dealt with FAPI much before, and it was a real eye-opener discovering I could script content type exports and imports easily in a few lines of code, and then further realizing I could actually script any form the same way. I started thinking about how many Drupal objects can be scripted quickly this way - node load/save, user load/save, any form, etc. It was then that I started thinking that maybe all we need to tackle the deployment problem is a) a way to abstract each of these pieces of data and b) a way to move them from server to server. Maybe Drupal doesn't need an overarching deployment solution, it just needs some glue.

I have written a proof-of-concept framework to work through some of these ideas. It is structured similarly to the Services module. Services provides a base core of functionality into which you can plug server modules (XMLRPC, JSON, REST, etc.) and service modules (node services, user services, etc.) Deployment also has three parts:

  • Deployment API - This implements the concept of a deployment plan. You create a deployment plan and add objects to it which will be deployed (content types, views, etc.) When the time comes these items are pushed to a server you specify via XMLRPC with an API key. The data stored about each item is extremely minimal, relying largely on the implementers to implement object-specific knowledge. Currently this is deploy.module
  • Deployment Implementers - Individual modules implement the deployment API to add the data they need to a deployment plan, and expose that ability to the front end. Currently I have content_copy_deploy.module and system_settings_deploy.module.
  • Deployment Services - Services modules which contain the knowledge to receive deployed data and do what is appropriate on the destination server. Currently I have content_copy_service.module and system_settings_service.module. One great thing about this is that regardless of what happens with the greater deployment framework, these services exist on their own as useful code which will be contributed back to the community.

So lets look at an example. Lets say I have a launch coming up where I will need to push live two content types, and pathauto information related to those content types. First I will create a new deployment plan, My Big Site Launch and indicate its destination URL and required API key (please note this API key will self-destruct before you ever read this.) This functionality is all provided through deploy.module.

Now I need to add my content types and settings to this plan. You can see below that there is a new tab on the content types listing page called deploy, and clicking it allows you to choose a content type and a deployment plan to add it it to.

This is all surfaced through content_type_deploy.module. Note there was no hacking of contrib or core modules in anything you see here. I have similarly exposed a deployment plan drop down on all forms that make use of system_settings_form through system_settings_deploy.module as you can see below in the pathauto settings screen. I'd like to acknowledge Daniel Kudwien and his Journal module here, as it was their code that provided the inspiration for how to handle system_settings_forms.

Having added my items to the plan, you can see a listing of them below.

On the destination server (specified above) I have enabled content_copy_service.module and system_settings_service.module.

Note the destination server doesn't even need deploy.module or any of its related modules at all! The only thing deploy.module offers is a way to collect data together about items to deploy, and a way to send them all out in one mouse-click. The services receive the data pushed forward by deploy.module and do with it whatever is needed.

So all I need to do now is click the Push button and everything will go on its way. I wanted to have a screencast to show the whole process to drive the point home, but I couldn't get it together in time. Hopefully after the weekend I can get all the pieces together to make that happen.

This framework is extremely powerful because it is a) modular and b) flexible. In my eyes, it is very drupal-like in all the best ways. Any piece of core or contrib could easily write their own deployment and services code and be in business to allow their data to be pushed around. Anyone who needs these services can either release their own modules to do it or even better submit patches to the modules in question to bring everything together. While I am mostly focused on the admin area, there is no reason why this framework couldn't be used to push content around as well. Services already includes node and user services which allows for saving and loading of those objects remotely.

While this is all exciting (well it is to me anyways) this is far from a complete solution right now. There are a great many gaps that need to be filled and questions to be answered:

  • Security - My original version of this code used xmlrpc.php, which was a disaster security-wise. Services includes API keys which helps, but I would want someone with a stronger background in security looking over this before I deemed it fit for use.
  • Dependencies - Do the appropriate modules live on the destination server? Are they the proper versions? Do all the nodereferenced content types exist?
  • Ordering - Related to the above, how to make sure deployment happens in the right order. If content type foo nodereferences content type bar, you have need to push out content type bar first. If pathauto has settings for content type foo and bar, they both need to go before the settings go. It would be nice if everyone knew this as well.
  • Versioning - There is no concept of versioning, is the content on my dev server newer or older than the content on my live server?
  • Rollback - There is no automated way to roll changes back, unless you count "take a mysqldump before you begin deployment, and if something goes wrong restore it."
  • Drupal 5 - This code is currently D5-only and thus in grave danger of becoming irrelevant sooner than later.

The holy grail would be integrating a full-featured module with SVN/CVS checkout and packaging, for full one-click scripted deployment.

So there's obviously a lot of work to be done, both on the above problems and the fact that the code even as it exists now needs some shoring up. Some of the above I have ideas about, some not so much. Hell, Workhabit might release Autopilot tomorrow and it will solve everyone's problems and render this whole conversation moot (although I suspect there will be room for multiple solutions in this sphere.) However, my plan right now is to spend a week or two getting this set of functionality into shape and release it as an alpha. I will probably release the services first, because they're the closest to being done.

I am actively interested in hearing what people think of this idea conceptually, and I'm also looking for collaborators on this project. The more brains the better. Let me know in the comments or the contact form!

I wrote two chapters of this book - Drupal 7 Module Development and I co-wrote it with Matt Butcher, Larry Garfield, Matt Farina, Ken Rickard, and John Wilkins. Go buy a copy!
I am the owner of the configuration management initiative for Drupal 8. You can follow this work at the dashboard on groups.drupal.org.

I used to work at NodeOne in Stockholm, Sweden. NodeOne is the largest pure Drupal consultancy in Europe. They have built websites for clients like IKEA, SFBio, and Möbler. If you need some work done get in touch!