Thursday, July 08, 2010

Keeping Hudson configuration and data in SVN

We recently lost our hudson server due to a multiple disk failure in the RAID array storing our hudson configuration. [5 of the 15 disks died]

So I've been looking into a backup script that will allow us to keep a backup of the configuration.  We use Maven for most of our builds, so the released artifacts are in our Maven Repository (which is hosted on two servers each with RAID arrays and using DRBD to mirror between the pair, with an rsync to a NAS in another cabinet and we are trying to get an rsych to an off-site storage going as well).

There seem to be two main options:
  1. Use the Hudson Backup Plugin.
  2. Backup your Hudson configuration to Source Control. 
Option 1 is nice, but you still have to copy off the backup files and find some safe place to store them.  Option 2 sounds better to me as if we loose our source control system, we'll there's nothing to build anyway!

So a quick search of the interwebs revealed Mike Rooney has been here before... cool... I have somewhere to start... with a few tweaks I have ended up with the following:
#!/bin/bash
if [[ ! -d $HUDSON_HOME ]]
then
    echo "Hudson home directory ($HUDSON_HOME) is missing or undefined"
    exit 1
fi

cd $HUDSON_HOME

# Add any new conf files, jobs, users, and content.
svn add -q --parents *.xml jobs/*/config.xml users/*/config.xml userContent/*

# Add the names of plugins so that we know what plugins we have
ls -l plugins > plugins.list
svn add -q -N --parents plugins.list

# Ignore things in the root we don't care about.
echo -e "war\nlog\n*.log\n*.tmp\n*.old\n*.bak\n*.jar\n*.json\nsecret.key\ntools\nshelvedProjects\n.owner\nupdates\nplugins" > myignores
svn ps -q svn:ignore -F myignores . && rm myignores

# Ignore things in jobs/* we don't care about.
echo -e "builds\nlast*\nnext*\n*.txt\n*.log\nworkspace*\ncobertura\njavadoc\nhtmlreports\nncover\ndoclinks" > myignores
svn ps -q svn:ignore -F myignores jobs/* && rm myignores

# Remove anything from SVN that no longer exists in Hudson.
svn st | sed -n -e "s/^\!//p" | xargs -r svn rm

# And finally, check in of course, showing status before and after for logging.
svn st && svn ci --non-interactive --username=hudson-build -m "automated commit of Hudson configuration" && svn st

The main changes are that I've updated the root level ignores and I capture a listing of the plugins directory so that you can know exactly what plugins you had installed
at

5 comments:

  1. MichaelWed Jul 14, 02:22:00 am GMT+1

    Glad my post helped, and thanks for the improvements. I like the idea of including the plugin list so I think I shall add that as well!

    I considered just keeping the plugins directory under VC as well as a lot of jobs may not work without installing them, but it can be rather large.

    ReplyDelete
  2. Harald WellmannMon Jul 19, 09:42:00 pm GMT+1

    Thanks, got to give it a try, either way. I didn't even know about the Backup Plugin.

    ReplyDelete
  3. meowsqueakTue Mar 08, 08:50:00 pm GMT

    Thanks for this - it inspired me to write a similar script for use with git (which interestingly turns out to be simpler syntactically).

    ReplyDelete
  4. UnknownFri Nov 23, 12:04:00 pm GMT

    also add:

    jobs/*/promotions/*/config.xml

    if you use the promotions plugin.

    ReplyDelete
  5. UnknownTue Nov 27, 11:36:00 am GMT

    svn st | sed -n -e "s/^\!//p" | xargs -r svn rm

    would fail if some dir/file contains spaces (think about a job named "test delete" that was removed)

    I'm using this instead:
    svn st | sed -n -e "s/^\(\!\s*\)\(.*\)/\"\2\"/p" | xargs -r svn rm

    ReplyDelete