ReleaseEngineering/How To/Land Buildbot Master Changes

From MozillaWiki
Jump to: navigation, search

This page is intended to aid you in understanding the repositories related to Mozilla Corporation Buildbot Masters and how to successfully apply patches to them.


  • buildbot - This repository is our local copy of the upstream Buildbot code. It contains imports of upstream releases and often patches that have not made it into an official Buildbot release. Our Buildbot masters and slaves use an installed version of the code from this repository.
  • buildbotcustom - This repository contains many custom Factorys, Steps, Schedulers, and other non-upstreamable Buildbot code. It generally contains code that is or can be shared between Buildbot masters.
  • tools - Our Buildbot masters also use some library functions from the build/tools Python library. Generally, this is code that is used by both Buildbot Masters as well as client-side scripts or other, non-Buildbot tools.
  • buildbot-configs - This repository contains Buildbot master configuration, including most branch specific things, mozconfigs, and other configuration details.


Some of the aforementioned repositories are managed with different in-repo, named branches. The table below should serve as a guide to understanding them:

Repository Branch Description
default This branch is the first point of landing for any Buildbot changes targeting the 0.8.x masters.
production-0.8 0.8.x Buildbot masters track this branch. Changes should not be merged to it until you are about to update the affected masters.
buildbot-configs default This branch is the first point of landing for any Buildbot changes targeting any masters.
production All Buildbot masters (regardless of Buildbot version) track this branch. Changes should not be merged to it until you are about to update all affected masters.
tools default Currently, this is the only used branch of the tools repository. It should be treated like the "production" branches of other repositories.

How to Land Things, Normally

This section works on the assumption that you have one or more reviewed and tested patches. When you do, the following procedure should be used to land them:

Land your patches

Land your patches on the appropriate "default" branch. For 0.8 masters this is always 'default'.

This can be done at any time. It does not require you to touch production Buildbot masters.

Watch for errors in Preproduction.

Our Preproduction systems run a variety of tests on landings to the default branches. They will send e-mail if any errors occur. Watch for these and backout or fix your patches appropriately.

Specifically, look at recent builds.

Merge your changes to the production branch

<p>When you are satisfied with the preproduction results and ready to land (ie: do the master reconfigs) you should merge your changes to the appropriate production branch. When doing so, you should be prepared to take any other tested changes from the default branch, too. See below for some helpful tips on merging.

Update the Masters

The masters are already set-up to track the production branches of the repositories. After you've pushed your changes to HG you can update the checkouts and reconfig/restart as normal. All masters which track the production branch you've merged must be updated. See ReleaseEngineering/Managing_Buildbot_with_Fabric for one technique to make this easier.

Update the Maintenance Page

The Maintenance page should be updated with the details of your landings.

Merging Tips

Previewing a Merge

To see which changesets will be merged without affecting your local repository, update to the production branch you plan to update, and run "hg merge -P". For example, to preview a merge from the 0.8 default branch in buildbotcustom, do the following:

hg up -r production-0.8
hg merge -P default


To perform the merge, use the same merge command as above, without "-P". For example:

hg up -r production-0.8
echo "Merging default -> production" > preview_changes.txt
hg merge -P default >> preview_changes.txt
hg merge default
hg commit -l preview_changes.txt


When pushing, be sure that you're pushing the correct branches. When working with multiple branches it's good to check 'hg out' to ensure you're only pushing what you intend to. If you have changes on other branches you do not want to push you can use "-b" to restrict which branches are pushed. For example, to only push changes on the default branch of buildbotcustom, use the following:

hg push -b default ssh://

You can also use "-b" as an argument to "hg out".

How to Land High Priority Things

Sometimes we don't have time to wait for preproduction or don't wish to pull in everything else on the default branch. If you're in a situation like this you can use the following procedure to jump the queue:

  • Commit your patch to the production branch and the default branch.
  • Push
  • Update the masters

This way, you've landed only your own patch, but still kept 'default' up to date.

Situations that call for this should be very rare.