ReleaseEngineering/Buildbot Best Practices

WIP

We've gotten to the point of having many different masters, and more complex masters. It's becoming more and more important to have consistent setup in terms of where configuration files are stored, where supporting code is stored, and other such details. This document is a rulebook and guideline for how to maintain these machines and instances.

Repositories

buildbot-configs - This repository contains Buildbot master.cfg and mozconfig files for most of our Buildbot instances.
buildbotcustom - This repository contains custom steps, factories, and other Mozilla specific Buildbot code required by our Buildbot masters.
buildbot - This repository is an import of upstream Buildbot code, plus some patches that we require.

Check-in Policies

We track configuration updates to our masters on BuildbotMasterChanges. Any change to a production Buildbot Master should be tracked on this page.
Patches should not be checked in until you are ready to update the master with them. This helps to avoid situations where an urgent fix needs to go in, and a random unrelated patch ends up getting enabled at the same time.
Production masters should never contain local changes. Even if you are just testing something you should check it in and pull it rather than making the change locally. Having temporary changes in the version control history is useful when debugging things later.
If you have a patch that affects multiple masters you should update all of the masters when you land your patch.
- This is less strict for staging masters. Eg, you shouldn't update a staging master if someone is currently using it.

How to land a patch

Below is one way to land a patch for a Buildbot master. Your technique may vary, but this will ensure there are no other patches sneaking in with yours, and no local changes. Before doing any of the below, land your patches into the repositories.

Look for local changes on the master.

cd /tools/buildbotcustom/buildbotcustom
hg diff
cd /builds/buildbot/configs
hg diff

Check to make sure only your patches are going to be enabled. If more than just your changeset show up in 'hg in' you can use 'hg in -p' to look at a diff of all of the patches that are going to be pulled in. It's not uncommon for other changesets to be there in buildbot-configs, since it holds configs for so many masters. Use your common sense here.

cd /tools/buildbotcustom/buildbotcustom
hg in
# If there are more than just your changeset use 'hg in -p' to look at the diffs
hg in -p
# Same thing for configs
cd /builds/buildbot/configs
hg in
hg in -p

Now, assuming there are no interfering patches, pull the changes for real

cd /tools/buildbotcustom/buildbotcustom
hg pull && hg up
cd /builds/buildbot/configs
hg pull && hg up

Run checkconfig to make sure the reconfig/restart will succeed

cd /builds/buildbot/$master
buildbot checkconfig

If everything looks OK, reconfig or restart the master

buildbot reconfig `pwd`
# OR
# You may need to run stop multiple times before Buildbot recognizes that the process is dead
# You can also check 'ps auxwww | grep buildbot' for it.
buildbot stop `pwd`
buildbot start `pwd`

Watch the Buildbot Waterfall for problems.