Lessons from Devblocks: Automating incremental upgrades in your PHP apps and their databases

(This post was inspired by this Twitter conversation: http://twitter.com/cakephp_dennis/status/1303663188)

Scenario:

You want to deploy copies of your PHP application that are outside your control (e.g. customers install the software on their own servers).  You want to make it easy for customers to stay current by having your application automatically patch its database schema and perform arbitrary upgrade tasks when it detects the project files have changed. This is especially simple when paired with a public repository (e.g. Subversion) that customers can use to grab the latest stable updates to your app.

The Devblocks Philosophy:

The approach I’m going to explain is based on how I implemented automatic database upgrades in Devblocks.  Devblocks is the PHP5 framework I’ve been building while rewriting Cerb4 from scratch over the past 2 years; and its philosophies are based on the development lessons (a.k.a. “hard knocks”) I’ve picked up from 7 years of working on the same project.  In a nutshell, it focuses on reusability of components (like any good framework) and making “change” as cheap as possible.  Even though Devblocks isn’t something I’m ready to support publicly yet, it’s currently powering thousands of copies of Cerb4.  Most of the Devblocks philosophy applies to building any kind of web application that strives to constantly evolve from ongoing user feedback.

The Nitty-Gritty:

You need a BUILD constant somewhere, which Subversion could even set into a particular file automatically; though it’s useful to set by hand during active development.

Your bootloader needs to compare the BUILD constant to a copy sitting in the cache somewhere.  When your files are swapped out this cached copy will persist with the previous BUILD version.  An update is triggered when the BUILD doesn’t match the cached copy or when the cached copy doesn’t exist.

You should have an /upgrade controller that users are automatically redirected to when the app detects a new version has been swapped in place of the previous files *and* that a database patch is necessary.  This would lock down the application to prevent any requests from being served until the patch is complete.  It should also kill existing sessions to prevent sessions being tied to stale data or objects.  You should have a security mechanism allow administrators to proceed with the upgrade (by IP, etc). The upgrade process also sets a lock file to ensure it’s not run simultaneously by two different administrators.  If the files change but the platform doesn’t have any pending database changes then there’s no reason for the app to lock itself, and users should be able to seamlessly continue with what they were doing.

In the case of Devblocks, everything is a plugin and each plugin has a manifest (in XML) that is independent of its code.  The /upgrade process reads these manifests to learn about new changes.  Devblocks has an “extension point” for “patch containers”, which means plugins can contribute their own scripts to be run by the platform during an upgrade.  A patch container can have multiple patches (e.g. 1.0, 1.1, 1.2, …) which are just PHP files.  The container associates a revision number with each patch.  Patches are run order when their revision number is greater than the last run revision for that patch.  You can reuse a single patch for all the database changes needed during development of a specific version or milestone.  You should always add the latest changes to the end of a patch and resist trying to group changes by table; as a day will come where you make a change, release it, and then change or revert it.  You need to handle the cases where someone (such as QA or other developers) had schema changes that you’re reverting.  Patch changes should always be in chronological order, top to bottom, even if it means you’re adding a change just to drop it a few lines later.  Trust me.

Example:
1.0 (revision 20)
1.1 (revision 105)
1.2 (revision 232)

The revisions for old patches will stop once that version is released and the revision will continue to increment on new files.  New files don’t reset the revision (it’s plugin-wide).  If a customer was upgrading from revision 100 of a hypothetical plugin, according to the example above they’d end up running the 1.1 and 1.2 patches  Each script is designed with the requirement that it needs to be capable of running multiple times, so patches check the schema before making changes (if column “X” doesn’t exist then add it; if column “Y” exists then drop it).  This means a customer can be at any past version and seamlessly upgrade to the latest version in one step.

Since everything in Devblocks is a plugin, the “core” plugin has its patch run first since all plugins are dependent on it.  Though it’s important that plugins only modify tables they created.  The platform ensures all patches are run in order and return successful status codes, and any errors will halt the process to prevent inconsistency.

This process is also really handy in development since you can run the /upgrade controller to make database changes to your local database and know you’re using the same code that customers ultimately will.  You won’t accidentally have tables and columns that you forget to commit.  It’s also useful during development for easily sharing changes with other developers (or QA) since the incremental diffs can be modified by multiple people and merged without incident.  Running /upgrade manually should ignore the version check and always try to run any pending updates.  When making changes in development you could increment the revision number of a patch without needing to change the BUILD of the app.  They have no need to stay in sync; the global BUILD triggers an automatic update, and customers will update far less often than developers.

When setting the app’s BUILD by hand, all you need to do is make sure you’re incrementing.  If you use Subversion you could commit with a build number one higher than the current revision (which your commit will use) or you can have Subversion automatically substitute the latest build number into the app on each commit using svn:keywords.

As an added tip, we also write all our resource URLs (images, scripts, etc) to the browser by appending the latest app BUILD as a query argument (like “script.js?v=BUILD”), which will force most browsers to recache your content every time the files are updated.

If there’s any interest, I can make a minimalistic Devblocks project to demonstrate this in action.  You could also download a free copy of Cerb4 and study the code.

-Jeff