Release:Release Automation on Mercurial:Documentation

Release Automation on Mercurial: Documentation | Plan | Details

Notes and Definitions

• 'Staging server' typically means stage.mozilla.org for production (read: real) releases and staging-stage.build.mozilla.org for staging (read: test) releases.
• 'Candidates directory' refers to the directory on the staging server which builds are deposited into. It almost always resolves to /home/ftp/pub/$product/nightly/$version-candidates/build$buildNumber - resolve the variables appropriately. For example: /home/ftp/pub/firefox/nightly/3.1b3-candidates/build1.
• Unless otherwise specified any references to directories on the staging server should be taken as relative to the candidates directory.
• Before starting any release, make sure that you clobber the appropriate release builders as they do not implicitly do so.

Overview

Mercurial Release Automation is a collection of Buildbot BuildFactory's which perform most of the tasks which are done as part of a release.

Code is stored in a few different places:

• BuildFactory's - buildbotcustom
• Configuration files - buildbot-configs
• Helper scripts - build/tools

Final Deliverables

The most important deliverables are the packages, installers, and updates. This includes (for each locale)

• Linux tarball, Mac dmg, Windows installer (exe)
• Complete and partial MAR file
• Update snippets for test and live channels

All of the above are handed off to and tested by QA during the release process.

Other, less important deliverables include:

• Source tarball and bundle
• Langpacks per platform per locale (minus en-US)
• Release branch and tags on appropriate repositories

Source packages are used by a small number of individuals and some Linux distributions to do their own builds of Firefox. Release branches and tags are very important to Release Automation and also used by developers when a release must be respun.

Individual steps may have additional deliverables which only serve to feed later steps. These will be discussed in the sections below.

Getting things going

Configuration

Where possible (and where it makes sense to), the release configuration reuses variables from the main config.py file. This file is imported as 'nightly_config' in release_master.py, so it's very easy to see when such a variable is used. Here's a complete list of the variables used from config.py:

Variable	Description
HGURL	The URL to the root of hgweb. Eg, http://hg.mozilla.org/
HGHOST	The plain hostname of the hg server. Eg, hg.mozilla.org
BUILD_TOOLS_REPO_PATH	The path, relative to HGHOST, where the build tools repository is located.
CONFIG_REPO_PATH	The path, relative to HGHOST, where the configuration file repository is located.
CONFIG_SUBDIR	The directory, relative to the root of the config repo, where platform specific mozconfig directories live.
STAGE_SERVER	The server to upload builds to.
STAGE_USERNAME	The username to use when uploading builds.
STAGE_GROUP	OBSOLETE/REMOVE ME
STAGE_BASE_PATH	OBSOLETE/REMOVE ME
STAGE_SSH_KEY	The ssh key to use when uploading builds. Needs to be in a platform independent format (eg, ~/.ssh/...)
AUS2_HOST	The host to upload update snippets to.
AUS2_USER	The username to use when uploading update snippets AUS2_HOST.

We also use a lot of platform specific variables (env, objdir) in the Build step that don't need to be called out here.

release_config.py contains variables which are specific to releases, and don't make sense to put in config.py. Things such as version numbers, config file names, and other things are stored here. Details on them below.

Variable	Description
hgUsername	The account name used when pushing tags, version bumps, and other configuration file updates back to repositories hosting on hg.mozilla.org.
hgSshKey	The path to the ssh key to used when pushing things back to hg.mozilla.org. The username in hgUsername must be setup to accept this key in order for the automation to work correctly. Path is generally in the ~/ form in order to be platform independent.
sourceRepoName	The name of the source repository to build from. Generally is the last part of sourceRepoPath, which is described below. This must be one of the branches listed in config.py
sourceRepoPath	The path, relative to the root of the hgHost, to the source repository to build from. http://hgHost/sourceRepoPath must be a cloneable Mercurial repository.
sourceRepoClonePath	This variable is only used in staging. This is the path to the source repository which will be cloned as part of the repo_setup build. All other parts of the automation use sourceRepoPath.
sourceRepoRevision	This is the revision which will be branched + version bumped from. It should be _exactly_ what is signed off on in release-drivers mail. It should also be updated when buildNumber is greater than 1 (ie respins).
relbranchOverride	This is used in two cases. Most commonly, you must pass it when buildNumber > 1, and should be the release branch which was generated for build1; use '' for build1. Less commonly, you can pass it to override the automatically generated relbranch name for a build1. Note, the default format is like this: GECKO191_YYYYMMDD_RELBRANCH, where YYYYMMDD is the current date in PST. Note that this is different than Bootstrap, when uses UTC.
l10nRepoClonePath	This is exactly like sourceRepoClonePath, but is the root of the l10n repository directory used for cloning locale repos in repo_setup.
l10nRepoPath	This is exactly like sourceRepoPath, but for locale repositories. http://hgHost/l10nRepoPath/ab-CD should be a cloneable repository.
l10nRevisionFile	This is the path, relative to the master's directory, which contains a listing of all of the locales to build, and their changesets. This file is read in at buildmaster start/reconfig time and the information is passed along to ReleaseTaggingFactory, which tags all of the repositories for this release. Any subsequent steps which require a list of locales will pull them from shipped-locales. This is a bug that may or may not be fixed.
cvsroot	The cvsroot to use for pulling MozBuild, patcher tools, and other things from CVS. It is also used to check in an update patcher config file and as such must be an account with write access.
productName	The lowercase product brand name. Eg, 'firefox', 'thunderbird', etc.
appName	The lowercase application name. Eg, 'browser', 'mailnews', etc.
version/appVersion	Sometimes we need the application version to be different from what we "call" the build, eg public release candidates for a major release (3.1 RC1). appVersion and oldAppVersion are optional definitions used in places that# don't care about what we call it. Eg, when version bumping we will bump to appVersion, not version. As a concrete example, when we produce the first release candidate for 3.1, version will be '3.1rc1' and appVersion will be '3.1'.
milestone	The milestone version of the current platform release. Eg, 1.9.1b2
buildNumber	The build number of the current release. 1 if it's the first try for this release. May be 2, 3, or higher if respins were required.
baseTag	The base tag to use when tagging repositories. _BUILD$buildNumber and _RELEASE will be appending to this where necessary.
oldVersion/oldAppVersion	Same thing as version/appVersion, but applies to the previous release. When 3.1rc1 is shipped oldVersion and oldAppVersion will both be 3.1b3. When we ship 3.1.1 oldVersion will be, eg, 3.1rc2 and oldAppVersion will be 3.1.
oldBuildNumber	The build number of the previous release. This should be the build number which was actually shipped.
oldBaseTag	Same as baseTag, but for the previous release.
releasePlatforms	This is a tuple containing all of the platforms to build as part of the release. These platforms must exist in config.py for the branch being built.
patcherConfig	The name of the patcher config file to bump and pass to patcher2.pl.
patcherToolsTag	The tag to update both the tools and source repository to before running patcher2.pl.
ftpServer	The ftp server to use for 'beta' channel snippets when useBetaChannel is '1'. Regardless of what useBetaChannel is set to this must be passed.
stagingServer	The stage server to use for 'betatest' snippets and most verification tests.
bouncerServer	The bouncer server to use for 'release' and 'releasetest' channel snippets.
ausServerUrl	TODO
releaseNotesUrl	URL of release notes. If None use the default URL. Introduced in bug 553059.
useBetaChannel	This variable controls which channel is used as the final release channel. When 0, 'beta' channel snippets will be considered the final release channel, and will point to mirrors. In this case, 'release' channel snippets will NOT be generated. When 1, 'release' channel snippets will be generated and will be considered the channel for final release. In this case, beta snippets will still be generated and will point to ftpServer.
*VerifyConfig	The configuration file names for the update verify configuration files for each platform.
majorUpdateRepoPath	The relative path repository that contains the shipped-locales file for the major update "to" release. If None, the rest of the majorUpdate* variables are not necessary.
majorUpdateToVersion, majorUpdateAppVersion, majorUpdateBuildNumber, majorUpdateBaseTag	These variables contain information about the release you are generating a major update to. See version/appVersion/buildNumber/baseTag for descriptions of them.
majorUpdateReleaseNotesUrl	Major Update release notes URL. If None use the default URL. Introduced in bug 553059.
majorUpdatePatcherConfig	The Patcher config file to use for the major update. This can be an existing file, in which case it will be overwritten and committed, or a non-existant file, in which case it will be created and committed.
majorUpdateVerifyConfigs	See updateVerifyConfigs description above.
testOlderPartials	Applicable only to the "updates" step. When true, the update verify bumping step will mark the n-2 release to have both its partial and complete updates tested. When false, it will rely on the default behaviour of update verify (As of bug 514040, this is "test complete only for n-2 and earlier builds). Generally, should be True on 1.9.1 based releases and earlier and False on anything more recent.

Starting the automation

Generally, the workflow for kicking off Release Automation is as follows:

1. Update l10n-changesets (normally performed by someone from the l10n team days ahead of time)
2. Update the correct configuration file. 1.9.11.9.2 mozilla-central
3. Check the release_config1.py and release_config2.py symlinks to make sure one of them points to the branch you're building.
4. Reconfigure the buildbot master
5. Clobber the release builders
6. Kick-off automation with the following command (see staging section if running on staging):

buildbot sendchange --username=bhearsum --master=localhost:9010 --branch=releases/mozilla-1.9.1 -m "Firefox 3.2a1build1" doit

Of course, replacing username, branch, and the message appropriately. Be sure not to leave out --branch.

Steps

Tag

Deliverables:

• Release branch and tags on source and l10n repositories

This steps creates a release branch and tags in both source and locale repositories. For example, if you started Release Automation for Firefox 3.1b3 on January 17th, 2009 (building out of the mozilla-1.9.1 repository) you would end with the following in mozilla-1.9.1 and all appropriate l10n repositories:

• GECKO191_20090117_RELBRANCH named branch
• FIREFOX_3_1b3_BUILD1 tag
• FIREFOX_3_1b3_RELEASE tag

For source repositories, after creating the branch Release Automation will perform a version bump on the source repository to remove the 'pre' strings that are normally in them. After commiting this revision it will be tagged with both of the FIREFOX_* tags and pushed back from whence it came. The 'mozillaCentralRevision' variable in release_config.py is used as the branch point.

For l10n repositories the exact revisions give in l10n-changesets are used as the branch point *and* tagged with the FIREFOX_* tags. They do not require a version bump.

This step takes approximately 10 minutes to complete.

Source

Deliverables:

• Source bundle and tarball uploaded to the staging server.

There's not much to be said about the Source step. It will deliver a ready-to-be-built tarball and a ready-to-be-used Mercurial bundle file. These should appear in the 'source' directory on the staging server.

This step takes approximately 10-15 minutes to complete.

Build

Deliverables:

• en-US build for each platform (tarball for Linux, dmg for Mac, exe and zip for Windows) uploaded to the staging server
• en-US complete MAR files for each platform uploaded to the staging server
• *_info.txt files containing BuildID uploaded to the staging server
• debugging symbols uploaded to the symbol server

Although Build is only listed as a single step it is one of the longer steps, and run on each platform. It is very much the same as regular depend and nightly builds with a few notable differences:

• Uses a different mozconfig file
• Codesighs is not run
• An update snippet is not created

This step takes approximately 45 minutes to complete on Linux, 45 minutes on Mac (xserve), 1h30min on Mac (mini), and 3h15min on Windows.

Repack

Deliverables (per locale):

• A build per platform
• A complete MAR file per platform
• A langpack (.xpi) file per platform

We don't have any sort of verification that every locale we need has been built so it's a good idea to watch this as it goes along, or scroll back on the waterfall when it's done and look for errors.

Repacks are done in a parallelized manner across many slaves. When the en-US build for a platform finishes Buildbot will queue up all of the l10n builds for that platform. As slaves become free they will take those jobs.

Since this step is parallelized across many slaves it can take anywhere from 20 minutes to 2 hours depending on the platform and availability of slaves.

Signing

This is a semi-automated process, and should be done with this document.

An FtpPoller watches for the signing log to appear on the ftp server and triggers the next group of steps.

L10n Verify

As the name implies this is strictly a verification step. For both the current and previous release it compares the strings in the localized build with the strings in the en-US build. This diff is dumped to a file. Once this is done for every locale for the current and previous release a metadiff of $locale.diff for the current and previous release is done. If that diff has no output, no strings have changed.

This step takes approximately 30 minutes to complete.

Log Analysis

There are a few common things to watch for:

• In the "compare current/previous"
  • Anything about being unable to unpack or mount a package
    • Usually indicates a corrupt package
    • On Mac, can mean that '7z' is not installed or not in PATH.
• In the "Only in..." log:
  • Locale(s) only in the previous release:
    • Locale failed to build (make sure the locale exists in the candidates directory)
    • Locale removed from shipped-locales (compare shipped-locales from previous and current release)
  • Locale(s) only in the current release:
    • Locale newly added to shipped-locales (compare shipped-locales from previous and current release)

The metadiff is a bit trickier and less clean cut. In a typical stable release (eg. 3.1.3) there are no string changes, the l10n metadiff log should be empty. During beta cycles there are lots of string changes, and therefore lots of output in the metadiff log. It's generally a good idea to have a look through them and make sure every locale has the same changes. When possible, it's good to track them down to a bug, to make sure they were intentional.

Update/snippet Generation

Deliverables:

• Updated patcher config file, checked into CVS
• Partial mar file per-platform per-locale
• Update snippets on test & live channels
• Snippet comparison log

For those familiar with Bootstrap's Updates step this step is pretty much exactly the same.

After checking out the necessary modules and tools this builder goes to work doing the following:

• Bumping & checking in a patcher config file
• Cloning sourceRepo & building update-packaging tools
• Creating & uploading partial MARs
• Creating & uploading update snippets
• When applicable, creating snippets for previous builds of the release
• Pushing test snippets live
• When applicable, doing a comparison of live channel snippets vs. releasetest

This step is generally pretty failsafe but in the event that it needs to be run a second time or restarted at later step it's necessary to comment out the patcher config bump step (see below for details on how). Until bug 466999 FIXED is resolved the patcher config file can only be bumped once.

Note that both the sourceRepo and the tools repo will be updated to the patcherToolsTag. If the update-packaging tools or patcher config script has been updated since the last release it is a good idea (and sometimes necessary) to create a new UPDATE_PACKAGING_R# tag before starting the release.

This step takes approximately 1 hour to complete.

Snippet Comparison Log Analysis

Once updates is completed there will be a log from the snippet comparison step. This step compares the contents of the releasetest channel snippets to the contents of the live channel (beta or release) snippets. There's a few things to note about this log:

• It will usually contain warnings about missing snippets for alphas and betas. These are expected and ignorable.
• When there are warnings about other snippets missing, a 'Warnings' log will be created and the step will be orange. These must be investigated and explained/fixed before any live snippets are pushed.
• If any snippets differ in contents between the test and live channels a 'Diffs' log will be created and the step will be orange. The Diffs log will list the files which differ; the full diffs can be viewed in the stdio log. Any differences must be investigated and explained/fixed before any live snippets are pushed.

Update Verify

This step verifies both the MAR files and does quick tests on the update snippets. Firstly, it tests both MAR files by applying them against the previous version, and then comparing the result against the current version's full package. Once this is done for every locale it then ensures that an update snippet is present for older versions (that is, older than the previous version), and that the complete MAR the snippet points to is downloadable. Note that it does NOT attempt to apply it.

This step takes approximately 15 minutes on Linux, 45 minutes on Mac, and 30 minutes on Windows.

Log Analysis

This step is pretty good at reporting errors. Most errors will turn the step red, and be pretty clearly visible in the log. Nonetheless it's a good idea to have a scan through the log for anything abnormal (searching for FAIL and WARN are helpful here).

Here's some common failures:

• "Binary files source/bin/freebl3.chk and target/bin/freebl3.chk differ" (or softokn3.chk) on win32 in the complete MAR
  • This is normal in the win32 logs and is caused by bug 404340. Once this bug is fixed it should go away.
• "CRC check failed" and/or "LoadSourceFile failed" on a partial MAR
  • This typically happens when the previous release's complete MAR differs from the complete package. Because partial MARs are generated by diffing the previous complete MAR with the current complete MAR the partial will fail the CRC check when this happens. There's nothing that can be done about this, because the problem is with an already-release-build. The correct thing to do here is inform release-drivers that there will not be partials available on this platform, trim them out of the candidates directory, and remove the partial snippets.
• "FAIL: partial from xxxxx wrong size", "FAIL: actual size: 0"
  • This is usually caused by missing MARs. You should first check to see if the MAR in question exists in the candidates directory. If it doesn't, go back and look at the logs to see if there was a problem generating or uploading it. If it does exist in the candidates directory make sure the permissions are set properly (ffxbld:firefox, 664).
• "FAIL: Could not download source xxx"
  • Usually indicates missing packages or bad permissions in the candidates directory. Take the same action as above.
• "FAIL: no partial update found for xxx" or "FAIL: no complete update found for xxx"
  • Check the logs from the previous step to make sure 'pushsnip' was run successfully. This error usually indicates that snippets weren't pushed.

Major Update & Verify

As part of any release that is on a less than current branch we generally create and push a Major Update as part of it. For example, when we did a 3.5.8 release we generated a major update for 3.5.8 -> 3.6 alongside it. Note that major updates happen as part of the "from" release, not the "to" release.

Configuration

Configuration for the major update sits alongside the rest of the release config. It shares many of its parameters with the rest of the automation but also requires some of its own. Parameters used exclusively by major update start with 'majorUpdate'. Below is a list of the parameters that could be used to generate a 3.5.8 -> 3.6 major update (unused vars omitted):

hgUsername = 'ffxbld'
hgSshKey = '~cltbld/.ssh/ffxbld_dsa'
sourceRepoPath = 'releases/mozilla-1.9.1'
productName = 'firefox'
appName = 'browser'
version = '3.5.8'
appVersion = version
baseTag = 'FIREFOX_3_5_8'
buildNumber = 1
cvsroot = ':ext:cltbld@cvs.mozilla.org:/cvsroot'
patcherToolsTag = 'UPDATE_PACKAGING_R10'
ftpServer = 'ftp.mozilla.org'
bouncerServer = 'download.mozilla.org'
stagingServer = 'stage-old.mozilla.org'
ausServerUrl = 'https://aus2.mozilla.org'
useBetaChannel = 1
majorUpdateRepoPath = 'releases/mozilla-1.9.2'
majorUpdateToVersion = '3.6rc2'
majorUpdateAppVersion = '3.6'
majorUpdateBaseTag = 'FIREFOX_3_6rc2'
majorUpdateReleaseNotesUrl = 'http://www.mozilla.com/%locale%/firefox/3.6/details/index.html'
majorUpdateBuildNumber = 1
majorUpdatePatcherConfig = 'moz191-branch-major-update-patcher2.cfg'
majorUpdateVerifyConfigs =  {'linux':  'moz191-firefox-linux-major.cfg',
                             'macosx': 'moz191-firefox-mac-major.cfg',
                             'win32':  'moz191-firefox-win32-major.cfg'}

Running it

Major update runs are not kicked off automatically because they depend on both the "to" and "from" releases being at a certain point. Once both of these releases successfully run their 'updates' and update verify factories you can use "Force Build" on the "from" releases' 'major_update' factory to start the process.

If either the "from" or "to" release respin at any point you must regenerate the major update. If the "to" release respins, majorUpdateBuildNumber needs to be updated.

At the end of the factory snippets will be uploaded to the AUS server and the test snippets will be pushed live. After that, the major update verify builders will be triggered.

Update Verify

Everything in the previous "update verify" section applies here.

Generally, major updates generate more harmless warnings than other releases. Analysis on these is done when testing a new major update and noted on release notes. When verifying a major update it's always helpful to look at the previous one's notes to find a list of the harmless warnings. If you find warnings that are NOT in the previous major update's notes they must be investigated.

It's rare to see major update verify builders go green because of all of the harmless warnings.

Final Verification

Once mirror uptake has reached 50 or 60% this builder can be started with 'Force Build' from the Buildbot waterfall.

This is a simple test which is run after mirrors are propagated to ensure there's no (major) problems with them. It does the same thing as the second part of Update Verify: for every locale/platform/version combination it checks for an update snippet, and then proceeds to download the mar file the snippet points to. The difference here is that Final Verification checks releasetest snippets - which point to mirrors instead just stage.mozilla.org.

The step will turn red if there are http codes other than 200 or 302, or if FAIL is found in the log. If it does this, you should have a look in the log, and find out which mirror is failing. It's a good idea to keep an eye on that mirror for awhile to make sure it finishes getting all of the files.

NOTE: You can wget the output and run this:

grep HTTP/1.0 text | grep -v 302 | grep -v 200

No matter what the status it's good to scan the log.

NOTE: A rule of thumb is to have 45k mirror uptake for stable releases and 10-20k for betas.

Common Problems & Resolutions

Restarting the automation from a specific point

WARNING: This can get tricky if you have to restart a specific platform's build or repack steps (as opposed to all of them). Often, it's easier to use Force Build and kick off downstream builders manually. See the next section for details on that.

In general, if you need to restart the automation from a specific builder you can do the following:

1. Add dummy factories for steps you don't want to run. For example, to start the automation from build, you would add the following between the build_factory definition and the build *builder* definition:

build_factory = BuildFactory(
...
)
from buildbot.process.factory import BuildFactory
from buildbot.steps.dummy import Dummy
build_factory = BuildFactory()
build_factory.addStep(Dummy())
builders.append({
...
})

This way, the dummy step will run, and downstream builders will be properly triggered. You may do this for as many builders as necessary.

Restarting failed builders without patching the config

It's often easier to use Force Build or buildbot sendchange to restart specific things than it is to patch the configs, check in, update the master, etc. If you need to restart tag, source, an en-US build, l10nverify, updates, or update_verify you can use "Force Build" from the Buildbot waterfall.

If you need to restart a single l10n build (that is, one locale) on a single platform you may also use "Force Build", but be sure to use en_revision, l10n_revision, and locale as properties. The revision properties should be set to _RELEASE tags, and the locale should be obvious.

If you need to restart many or all locales you'll have to fill out the web interface multiple times, until bug 518589 is resolved.

Restarting a builder from a certain point

In some cases only part of a Builder will successfully complete. For example, if the 'updates' builder was able to build partial MARs successfully but failed to upload them you will want to figure out what the problem is, and then restart it from the upload step. To do that, open up /tools/buildbotcustom/buildbotcustom/process/factory.py and comment out all steps before uploading. You should also find out which slave it was started on, and update the 'slavenames' for 'updates' in release_master.py to ensure the job gets started on the same slave.

Depending on how much of a builder is left to run it can be faster and/or easier to simply run the steps manually and continue on. For example, if backupsnip or pushsnip in 'updates' failed, it's probably easier to run those steps manually rather than go to the trouble of starting another job.

In other cases, it may make sense just to start from the beginning. Eg, if the build tools repository fails to clone, it's generally easier just to start from scratch.

Use your judgment here.

Tagging failed out part way through

If this happens, you'll need to continue it on but get it to skip locales which it has already tagged. To do so, delete any locales which have been tagged from l10n-changesets and reconfig. If the source repository has already been tagged you should pass 'l10n_repositories' to ReleaseTaggingFactory instead of 'repositories. Note that the rest of the release automation uses shipped-locales so removing things from l10n-changesets doesn't cause locales not to build.

Need to rebuild only some locales

You can use Ben's force build scripts to do this. They require buildbotcustom. Full documentation and examples are provided with the script, but here's a quick tutorial (assumes you're running on a master):

# can be run from anywhere on mpt-vpn. a master, your laptop, etc
hg clone http://hg.mozilla.org/build/tools
cd tools/buildbot-helpers
# this example will retrigger all l10n for all platforms for 3.5.1
python force_release_l10n.py -m http://localhost:8010 -t FIREFOX_3_5_1_RELEASE -v -b releases/mozilla-1.9.1 -n [your name]

If you need to retrigger one platform at a time you can use -p, like so:

# linux only
python force_release_l10n.py -m http://localhost:8010 -t FIREFOX_3_5_1_RELEASE -v -b releases/mozilla-1.9.1 -p linux -n [your name]

You can also trigger two platforms at the same time with multiple -p arguments:

# windows *and* mac
python force_release_l10n.py -m http://localhost:8010 -t FIREFOX_3_5_1_RELEASE -v -b releases/mozilla-1.9.1 -p macosx -p win32 -n [your name]

If you need to trigger odder combinations of locales / platforms you need to construct your own shipped-locales file. For example, to restart af on linux, zh-TW on mac, and uk on all platforms you would put this in a file:

af linux
uk
zh-TW osx

And then trigger the script like so (note the use of -s instead of -b):

python force_release_l10n.py -m http://localhost:8010 -t FIREFOX_3_5_1_RELEASE -v -s locales-file -n [your name]

That's it!

Staging Specific Notes

Release automation in staging is mostly the same as in production, but does have a few differences you should known about:

• All uploading (builds, snippets, symbols) is done to staging-stage.build.mozilla.org
• It can take a few tries to get repo_setup to run properly. This is because hgweb sometimes returns a 500 (internal server error) when we query about a locale. The best solution is just to start the automation from scratch until it works, to make sure you get a clean run. If this is too frustrating for you, you can manually clone the repositories you care about and start automation from tag (see Restarting from a specific point).
• Staging doesn't have a lot of slaves, and you may need to go around and stop running builds to get your automation run to happen in a reasonable period of time.
• The branch to pass to the initial sendchange should match the sourceRepoPath in the release config. e.g.

  buildbot sendchange --user catlee --branch users/stage-ffxbld/mozilla-1.9.2  --master localhost:9010 "make it so"

Staging specific preparation

NOTE: Don't forget to clobber! (disconnect from Build-VPN)

http://build.mozilla.org/stage-clobberer/index.php

NOTE: It is highly recommended that you reduce the number of locales of this step to speed things up. Use this command to indicate which locales to exclude and make it part of the "wget -r -np blah blah" step (edit the output to match l10n-changesets_mozilla-blah.py):

for line in `wget -qO- http://mxr.mozilla.org/mozilla-central/source/browser/locales/shipped-locales?raw=1`; do echo "--exclude=$line" >> temp; done

Because we point the staging releases at staging-stage.build.mozilla.org you need to download the previous release to it. You can use the following handly shell script to do so. Note that the variables are for the *previous* release, not the one you will be running:

# ffxbld @ staging-stage
export VERSION=3.5.3
export BUILD=1
export PRODUCT=firefox
cd /home/ftp/pub/$PRODUCT/nightly
mkdir -p $VERSION-candidates/build$BUILD
cd $VERSION-candidates/build$BUILD
wget -r -np -nH --cut-dirs=6 -R index.html* http://stage.mozilla.org/pub/mozilla.org/$PRODUCT/nightly/$VERSION-candidates/build$BUILD/
cd /home/ftp/pub/$PRODUCT/releases
ln -s /home/ftp/pub/$PRODUCT/nightly/$VERSION-candidates/build$BUILD $VERSION

If you're doing a testrun with a limited number of locales you may delete any locales you don't care about after the above script finishes. (or you can do --exclude=af, etc for each unwanted locale)

Doing a test run with a limited number of locales

To run a test with a limited number of locales, do the following:

1. Locally modify l10n-changesets to include whichever locales you want.
2. Reconfig Buildbot and start the automation as normal
3. Once the tag builder is finished and before the end of the first en-US build, clone the staging source repository (eg, http://hg.mozilla.org/users/stage-ffxbld/mozilla-1.9.1) do the following:

hg up -C GECKO191_20090115_RELBRANCH # replace the relbranch appropriately, of course
# modify browser/locales/shipped-locales to include the same locales as l10n-changesets
hg commit -m "Reduce number of locales for this test run"
hg tag -f FIREFOX_3_1b3_RELEASE # using the right version number in the tag
hg push ssh://hg.mozilla.org/users/stage-ffxbld/mozilla-1.9.1

NOTE: You can do this from the slave that did the tagging to save time when checking out.

[cltbld@linux-ix-slave02 mozilla-central]$ vi browser/locales/shipped-locales 
[cltbld@linux-ix-slave02 mozilla-central]$ hg commit -u stage-ffxbld -m "Reduce locales"
[cltbld@linux-ix-slave02 mozilla-central]$ hg id
225b6e60404d (GECKO20b7_20101008_RELBRANCH) tip
[cltbld@linux-ix-slave02 mozilla-central]$ hg tag -u stage-ffxbld -f -r 225b6e60404d -m "Tag" FIREFOX_4_0b7_BUILD1
[cltbld@linux-ix-slave02 mozilla-central]$ hg tag -u stage-ffxbld -f -r 225b6e60404d -m "Tag" FIREFOX_4_0b7_RELEASE
[cltbld@linux-ix-slave02 mozilla-central]$ hg push -e "ssh -l stage-ffxbld -i ~cltbld/.ssh/ffxbld_dsa" -f ssh://hg.mozilla.org/users/stage-ffxbld/mozilla-central

How to "sign" in staging

We do not test signing as a part of a staging run. Mostly because it takes so damn long. Nonetheless, we need to shuffle files around so post-signing steps can find them. To do this, log onto staging-stage.build.mozilla.org and do the following:

# ffxbld@staging-stage
VERSION=3.5rc1
BUILD=1
cd /home/ftp/pub/firefox/nightly/${VERSION}-candidates/build${BUILD}
mkdir win32 update/win32
rsync -av --exclude=*.zip unsigned/win32/ win32/
rsync -av unsigned/update/win32/ update/win32/
rsync -av unsigned/win32_info.txt .
echo "faked" > win32_signing_build${BUILD}.log

We purposely make copies here rather than symlinking for a couple of reasons: L10n verify scripts barf when they get zip files (hence the --exclude above), 'updates' factory will blow away complete MARs upon upload if update/win32 is a symlink. The echo creates the log the automation looks for, in order to continue to l10n verify and updates.

Creating a CVS mirror for patcher and configs

If you need a new or modified patcher config, which shouldn't be checked into production CVS, then modify the following method:

WHO=yournamehere
# cltbld@staging-stage
cd /builds/cvsmirrors
mkdir -p ${WHO}/cvsroot.clean/mozilla/tools/
rsync -av --exclude=CVSROOT/config --exclude=CVSROOT/loginfo cvs-mirror.mozilla.org::mozilla/CVSROOT /builds/cvsmirrors/${WHO}/cvsroot.clean/
rsync -av cvs-mirror.mozilla.org::mozilla/mozilla/tools/patcher-configs /builds/cvsmirrors/${WHO}/cvsroot.clean/mozilla/tools/
rsync -av cvs-mirror.mozilla.org::mozilla/mozilla/tools/patcher /builds/cvsmirrors/${WHO}/cvsroot.clean/mozilla/tools/
rsync -av cvs-mirror.mozilla.org::mozilla/mozilla/tools/release /builds/cvsmirrors/${WHO}/cvsroot.clean/mozilla/tools/
rsync -a --delete-after /builds/cvsmirrors/${WHO}/cvsroot{.clean,}/

To make changes check out using

cvs -d staging-stage.build.mozilla.org:/builds/cvsmirrors/${WHO}/cvsroot co mozilla/

and specify a cvsroot of staging-stage.build.mozilla.org:/builds/cvsmirrors/${WHO}/cvsroot in the config for the release automation.