View source for Release:Release Automation on Mercurial:Documentation

== Notes and Definitions ==
* 'Staging server' typically means stage.mozilla.org for production (read: real) releases and staging-stage.build.mozilla.org for staging (read: test) releases.
* 'Candidates directory' refers to the directory on the staging server which builds are deposited into. It almost always resolves to /home/ftp/pub/$product/nightly/$version-candidates/build$buildNumber - resolve the variables appropriately. For example: /home/ftp/pub/firefox/nightly/3.1b3-candidates/build1.
* Unless otherwise specified any references to directories on the staging server should be taken as relative to the candidates directory.

== Overview ==
Mercurial Release Automation is a collection of Buildbot BuildFactory's which perform most of the tasks which are done as part of a release.

Code is stored in a few different places:
* BuildFactory's - [http://mxr.mozilla.org/mozilla/source/tools/buildbotcustom/ buildbotcustom]
* Configuration files - [http://hg.mozilla.org/build/buildbot-configs/ buildbot-configs]
* Helper scripts - [http://hg.mozilla.org/build/tools build/tools]
* [http://mxr.mozilla.org/mozilla/source/tools CVS]
* Mozharness - [http://hg.mozilla.org/build/mozharness build/mozharness]

Of particular note is [http://hg.mozilla.org/build/buildbotcustom/file/tip/process/release.py process/release.py], which instantiates all of the Schedulers, Builders, ChangeSources, and Status modules that releases use.

All steps are affected or fed in some way by the configuration files, but not all of them use code from the other locations. The subsequent sections contain links to the code that each step relies on.

== Final Deliverables ==
The most important deliverables are the packages, installers, and updates. This includes (for each locale)
* Desktop (Firefox)
** Linux/Linux64 tarball, Mac dmg, Windows installer (exe)
** Complete and partial updates for all desktop builds/locales
** Update snippets for test and live channels
* Mobile (Fennec)
** Mobile packages: Android APK, Maemo5-gtk DEB
** Mobile desktop: Linux tarball, Mac dmg, Windows installer (exe)

All of the above are handed off to and tested by QA during the release process.

Other, less important deliverables include:
* Source tarball and bundle for desktop (Firefox) and mobile (Fennec)
* Langpacks per platform per locale for desktop (Firefox) and mobile desktop platforms
* Release branch and tags on appropriate repositories
* Checksums for released files

Source packages are used by a small number of individuals and some Linux distributions to do their own builds of Firefox. Release branches and tags are very important to Release Automation and also used by developers when a release must be respun.

Individual steps may have additional deliverables which only serve to feed later steps. These will be discussed in the sections below.

== Getting things going ==
=== Configuration ===
Where possible (and where it makes sense to), the release configuration reuses variables from the branches' dep/nightly config. These will not be discussed here. [http://tinyurl.com/3yvwv6l This MXR query] will show you all of the places in the release automation set-up that branch variables are referenced.

The release configs themselves have a number of variables and options in them. These are discussed below.
==== sourceRepositories ====
This variable is information dense dictionary and mostly contains information only relevant to Tagging.
Each key of sourceRepositories (eg. "mozilla"), has a value of a dict with the following structure:
* name - The name of the source repository, without any leading directories
* clonePath - (staging only) Where to clone this repository from in repo_setup
* path - Where to clone this repository for everything else (tagging, builds, repacks, et. al)
* revision - The revision to base this release on. If relbranch is None or the provided one does not exist, the automation will update to this revision before creating the relbranch.
* relbranch - The name of the relbranch to use for this release. If set to None, it will be generated automatically. If provided and exists, the automation will update to it before doing any bumping or tagging. If provided and doesn't exist, it will be created.
* bumpFiles - This is another dict, whose keys are the paths in the sourceRepo, and whose values are *another* dict, with the following structure.
** version - The version number to use in this file on the relbranch
** nextVersion - The version number to use in this file on the default branch

Here's a concrete example of sourceRepositories for a Firefox build. Note that the bump file versions are typically factored out to earlier variables to avoid repetition.
<pre>
releaseConfig['sourceRepositories'] = {
  'mozilla': {
    'name': 'mozilla-central',
    'path': 'mozilla-central',
    'revision': 'aaaaabbbbcccc',
    'relbranch': None
    'bumpFiles': {
      'browser/config/version.txt': {
        'version': releaseConfig['appVersion'],
        'nextVersion': releaseConfig['nextAppVersion'],
      },
      'config/milestone.txt': {
        'version': releaseConfig['milestone'],
        'nextVersion': releaseConfig['nextMilestone'],
      },
    }
  }
}
</pre>

==== Others ====
{| class="fullwidth-table"
| '''Variable'''
| '''Description'''
|-
| hgUsername
| The account name used when pushing tags, version bumps, and other configuration file updates back to repositories hosting on hg.mozilla.org.
|-
| hgSshKey
| The path to the ssh key to used when pushing things back to hg.mozilla.org. The username in hgUsername must be setup to accept this key in order for the automation to work correctly. Path is generally in the ~/ form in order to be platform independent.
|-
| l10nRepoClonePath
| This is exactly like sourceRepoClonePath, but is the root of the l10n repository directory used for cloning locale repos in repo_setup.
|-
| l10nRepoPath
| This is exactly like sourceRepoPath, but for locale repositories. http://hgHost/l10nRepoPath/ab-CD should be a cloneable repository.
|-
| l10nRevisionFile
| This is the path, relative to the master's directory, which contains a listing of all of the locales to build, and their changesets. This file is read in at buildmaster start/reconfig time and the information is passed along to ReleaseTaggingFactory, which tags all of the repositories for this release. Any subsequent steps which require a list of locales will pull them from shipped-locales. This is a bug that may or may not be fixed.
|-
| l10nRelbranch
| The relbranch to use for l10n repositories. See "relbranch" in the preceding section for mechanics. Generally should be set to the same thing as the sourceRepositories relbranches.
|-
| shippedLocalesPath
| The path, relative to the root of the sourceRepo, to shipped-locales.
|-
| mergeLocales
| When True, merging will be enable when running compare-locales.
|-
| l10nChunks
| The number of l10n builders to be used, per platform. If not present, defaults to DEFAULT_PARALLELIZATION set in [http://hg.mozilla.org/build/buildbotcustom/file/tip/process/release.py process/release.py].
|-
| cvsroot
| The cvsroot to use for pulling MozBuild, patcher tools, and other things from CVS. It is also used to check in an update patcher config file and as such must be an account with write access.
|-
| productName
| The lowercase product brand name. Eg, 'firefox', 'thunderbird', etc.
|-
| appName
| The lowercase application name. Eg, 'browser', 'mailnews', etc.
|-
| binaryName
| The name which will be used in the filenames of installers, packages, and MARs. Usually is the same as productName.
|-
| oldBinaryName
| Same as above, but for the oldVersion.
|-
| version/appVersion
| Sometimes we need the application version to be different from what we "call" the build, eg public release candidates for a major release (3.1 RC1). appVersion and oldAppVersion are optional definitions used in places that# don't care about what we call it. Eg, when version bumping we will bump to appVersion, not version. As a concrete example, when we produce the first release candidate for 3.1, version will be '3.1rc1' and appVersion will be '3.1'.
|-
| milestone
| The milestone version of the current platform release. Eg, 1.9.1b2
|-
| nextAppVersion/nextMilestone
| These variables control what the appVersion and milestone will be bumped to on the "default" branch.
|-
| buildNumber
| The build number of the current release. 1 if it's the first try for this release. May be 2, 3, or higher if respins were required.
|-
| baseTag
| The base tag to use when tagging repositories. _BUILD$buildNumber and _RELEASE will be appending to this where necessary.
|-
| oldVersion/oldAppVersion
| Same thing as version/appVersion, but applies to the previous release. When 3.1rc1 is shipped oldVersion and oldAppVersion will both be 3.1b3. When we ship 3.1.1 oldVersion will be, eg, 3.1rc2 and oldAppVersion will be 3.1.
|-
| oldBuildNumber
| The build number of the previous release. This should be the build number which was actually shipped.
|-
| oldBaseTag
| Same as baseTag, but for the previous release.
|-
| enUSPlatforms
| This is a tuple containing all of the platforms that require an en-US build as part of the release. These platforms must exist in config.py for the branch being built.
|-
| l10nPlatforms
| A tuple containing all of the platforms that require l10n repacks. Any platform listed here must also be listed in enUSPlatforms.
|-
| talosTestPlatforms
| A tuple containing all of the platforms that require talos test runs to be requested. Platforms listed here must also be listed in enUSPlatforms.
|-
| unittestPlatforms
| A tuple containing all of the platforms that require unit test runs to be requested. Platforms listed here must also be listed in enUSPlatforms.
|-
| xulrunnerPlatforms
| A tuple containing all of the platforms that require XULRunner builds to be generated.
|-
| patcherConfig
| The name of the patcher config file to bump and pass to patcher2.pl.
|-
| patcherToolsTag
| The tag to update both the tools and source repository to before running patcher2.pl.
|-
| ftpServer
| The ftp server to use for 'beta' channel snippets when useBetaChannel is '1'. Regardless of what useBetaChannel is set to this must be passed.
|-
| stagingServer
| The stage server to use for 'betatest' snippets and most verification tests.
|-
| bouncerServer
| The bouncer server to use for 'release' and 'releasetest' channel snippets.
|-
| ausServerUrl
| The AUS URL to aus for update verify tests.
|-
| ausUser
| The username to use when uploading snippets to the AUS server
|-
| ausSshKey
| The SSH key to use when uploading snippets to the AUS server
|-
| releaseNotesUrl
| URL of release notes. If None use the default URL. Introduced in {{bug|553059}}.
|-
| useBetaChannel
| This variable controls which channel is used as the final release channel. In order to not generate 'release' channel snippets (ie: for beta releases) set it to 0 so that the 'beta' channel snippets will be considered the final release channel, pointing to mirrors. When set to 1, 'release' channel snippets will be generated and will be considered the channel for final release, pointing to mirrors and if the patcher config has 'beta' channel references, beta snippets will be generated and will point to ftpServer, otherwise no beta snippets are generated.
|-
| *VerifyConfig
| The configuration file names for the update verify configuration files for each platform.
|-
| updateVerifyChunks
| The number of builders to be used for update verify, per platform. If not present, defaults to DEFAULT_PARALLELIZATION set in [http://hg.mozilla.org/build/buildbotcustom/file/tip/process/release.py process/release.py].
|-
| majorUpdateRepoPath
| The relative path repository that contains the shipped-locales file for the major update "to" release. If None, the rest of the majorUpdate* variables are not necessary.
|-
| majorUpdateToVersion, majorUpdateAppVersion, majorUpdateBuildNumber, majorUpdateBaseTag
| These variables contain information about the release you are generating a major update to. See version/appVersion/buildNumber/baseTag for descriptions of them. 
|-
| majorUpdateReleaseNotesUrl
| Major Update release notes URL. If None use the default URL. Introduced in {{bug|553059}}.
|-
| majorUpdatePatcherConfig
| The Patcher config file to use for the major update. This can be an existing file, in which case it will be overwritten and committed, or a non-existant file, in which case it will be created and committed.
|-
| majorUpdateVerifyConfigs
| See updateVerifyConfigs description above.
|-
| testOlderPartials
| Applicable only to the "updates" step. When true, the update verify bumping step will mark the n-2 release to have both its partial and complete updates tested. When false, it will rely on the default behaviour of update verify (As of bug 514040, this is "test complete only for n-2 and earlier builds). Generally, should be True on 1.9.1 based releases and earlier and False on anything more recent.
|-
| doPartnerRepacks
| Whether or not to create and schedule partner repack builds as part of this release.
|-
| partnersRepoPath
| The repoPath of the repository containing the partner repack scripts and configs. Generally set to 'build/partner-repacks'. Only necessary when doPartnerRepacks is True.
|-
| AllRecipients
| List of email addresses to send step-by-step email notification of release progress, Defaults to release@
|-
| PassRecipients
| List of email addresses to send step-by-step email notification of release progress if and only if the step is successful, Defaults to release@, although we may want to send directly to r-d@
|-
| ReleaseTemplates
| Location of the message bodies to use to formulate the messages sent for the notifications
|-
| tuxedoConfig
| A configuration file of [http://hg.mozilla.org/build/tools/file/tip/release/tuxedo-add.py tuxedo-add.py], which contains FTP path templates for the deliverables (installers and MARs). Examples: [http://hg.mozilla.org/build/tools/file/620cb53444a7/release/firefox-tuxedo.ini stable version], [http://hg.mozilla.org/build/tools/file/620cb53444a7/release/firefox-devpreview-tuxedo.ini alpha (devpreview) version].
|-
| tuxedoServerUrl
| Tuxedo server's API URL prefix. Trailing slash is important.
|}

=== Preparation ===
There are a couple of things you can do prior to the "go to build" to hasten things when you do receive. Generally, we do the following ahead of time:
* Post updated configs and any other patches you'll need for review, using 'FILLMEOUT' or similar as the revision.
* Mark a clobberer for "Any master", $branch, "Any builder" with [https://build.mozilla.org/clobberer/ clobberer]. This will instruct any slave that did the last release on $branch to clobberer their release directories.

If you are doing a test release in staging there is [[Release:Release_Automation_on_Mercurial:Documentation#Staging_Specific_Notes|additional setup work]] to do, and the clobberer URL changes.

=== L10N Changesets ===
Currently we have separate changesets files for Fennec and Firefox. Both are generated from the [https://l10n-stage-sj.mozilla.org/shipping/milestones l10n dashboard]<br />
First log in to the dashboard with your LDAP and then follow the instructions below
====Fennec====
* Click "ship it" to load up the milestone (eg: Fennec 8 Beta Build 2)
* This will take you to a page like [https://l10n-stage-sj.mozilla.org/shipping/confirm-ship?ms=fennec8_beta_b2 https://l10n-stage-sj.mozilla.org/shipping/confirm-ship?ms=fennec8_beta_b2]
* For platforms field put
 linux,macosx,win32
* In the 'Add a multi-locale file for' field put
 android-multilocale
* select "Add"
* select "Generate it"
* Copy and paste it into your repo's l10n-changesets_mobile-{version}.json
* go back to the original page
* select "Ship it" to preserve the record of what was shipped

====Firefox====
* Click "ship it" to load up the milestone (eg: Firefox 8 Beta Build 2)
* This will take you to [https://l10n-stage-sj.mozilla.org/shipping/confirm-ship?ms=fx8_beta_b2 https://l10n-stage-sj.mozilla.org/shipping/confirm-ship?ms=fx8_beta_b2]
* Open in a new tab the "l10n-changesets" URL (don't touch any of the other fields)
* copy and paste the list into your l10n-changesets_mozilla-{version}
* go back to the original page
* select "Ship it"

=== Starting the automation ===
Generally, the workflow for kicking off Release Automation is as follows:
* Update the configuration files
* Tag buildbotcustom, buildbot-configs, tools, mozharness (for combined and Fennec-only releases) with the correct _RELEASE and _BUILDn tags. You must use FIREFOX_${VERSION}_BUILDn and FIREFOX_${VERSION}_RELEASE for Firefox releases, FENNEC_${VERSION}_BUILDn and FENNEC_${VERSION}_RELEASE for Fennec releases, or all 4 tags for combined releases.
* Update the Buildbot Master doing the release
* Run the pre-flight sanity check. This is done in the master directory (watch out for PYTHONPATH). For example:
 # Combined release
 python ../tools/buildbot-helpers/release_sanity.py -u bhearsum -V 6.0b2 --branch mozilla-beta --build-number 1 \
     --release-config release-firefox-mozilla-beta.py --release-config release-fennec-mozilla-beta.py --products firefox,fennec  \
     --dryrun localhost:9001
 # Firefox-only release
 python ../tools/buildbot-helpers/release_sanity.py -u bhearsum -V 6.0b2 --branch mozilla-beta --build-number 1 \
  --release-config release-firefox-mozilla-beta.py --products firefox  \
  --dryrun localhost:9001
* Start the automation by running the sanity script again, without --dryrun. For example:
 # buildbot should in in $PATH, source ve env
 source ../bin/activate
 PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u rail \
  -V 6.0b2 --branch mozilla-beta --build-number 1 \
  -c release-firefox-mozilla-beta.py -c release-fennec-mozilla-beta.py --products firefox,fennec localhost:9001


If you're working in staging you must make sure to pass in the correct staging release config (staging_release-firefox-<branch name>.py)

If for some reason you need to perform the sendchange manually, here's a sample. Note that the branch name is the repo path to the source repository. In staging, this will be a user repo. All parameters are required:</p>
 buildbot sendchange --username bhearsum --master localhost:9010 --branch releases/mozilla-release \
     -p script_repo_revision:FIREFOX_3_6_12_RELEASE -p products:firefox,fennec release_build

==== Starting Signing ====
The next step is to start the signing automation, which will download builds as they become available, and start the signing process once all builds are ready. For more details on starting the signing automation, see the
[https://intranet.mozilla.org/Build:CombinedSigning Combined Signing page].

Android signing procedure is outlined [https://intranet.mozilla.org/Build:MobileSigning here]

== Steps ==
=== Sync mozilla-beta to mozilla-release ===
Mercurial repositories used for final releases should be synced from mozilla-beta. 

Tip of default on mozilla-release should be tagged and closed. mozilla-beta should be tagged and pulled. It creates new tip of default on mozilla-release. See [http://mozilla.github.com/process-releases/draft/development_specifics/index.html#branching Aurora specific sync mechanics] for the details.

L10N repositories uses noop (dummy) merge if needed to simplify localizers life.



To handle this process use the following scripts.

==== Sync mozilla-beta to mozilla-release ====
<pre>
# Adjust VERSION variable which stands for the current Firefox version in mozilla-release
VERSION=8
HG_USER="ffxbld <release@mozilla.com>"

hg clone http://hg.mozilla.org/releases/mozilla-release
hg clone http://hg.mozilla.org/releases/mozilla-beta

beta_rev=$(hg -R mozilla-beta id -i -r default)
release_rev=$(hg -R mozilla-release id -i -r default)

RELEASE_BASE_TAG="RELEASE_BASE_`date +%Y%m%d`"
RELEASE_TAG="FIREFOX_RELEASE_$VERSION"

hg -R mozilla-beta tag -r $beta_rev -u "$HG_USER" -m "Added tag $RELEASE_BASE_TAG for changeset $beta_rev. CLOSED TREE a=release" $RELEASE_BASE_TAG
hg -R mozilla-beta push -e "ssh -l ffxbld -i ~/.ssh/ffxbld_dsa" ssh://hg.mozilla.org/releases/mozilla-beta

hg -R mozilla-release tag -r $release_rev -u "$HG_USER" -m "Added tag $RELEASE_TAG for changeset $release_rev. CLOSED TREE a=release" $RELEASE_TAG
hg -R mozilla-release commit -u "$HG_USER" -m "Closing old head. CLOSED TREE a=release" --close-branch

hg -R mozilla-release pull mozilla-beta
hg -R mozilla-release up -C default

# edit shipped-locales file if you need to remove some beta locales

# push back
# hg -R mozilla-release push -f -e "ssh -l ffxbld -i ~/.ssh/ffxbld_dsa" ssh://hg.mozilla.org/releases/mozilla-release
</pre>

==== L10N sync ====
<pre>
#!/bin/bash
set -x
set -e

HG_USER="ffxbld <release@mozilla.com>"
HG_HOST=hg.mozilla.org
repo=mozilla-release
release_repo_path=releases/l10n/mozilla-release
beta_repo_path=releases/l10n/mozilla-beta
wd=`pwd`/l10n

mkdir -p $wd
cd $wd

for l in `wget -q -O- http://$HG_HOST/releases/$repo/raw-file/default/browser/locales/shipped-locales | grep -v en-US | awk '{print $1}'`; do
    hg clone http://$HG_HOST/$release_repo_path/$l
    hg clone http://$HG_HOST/$beta_repo_path/$l $l.beta
    release_rev=`hg -R $l id -i -r default`
    beta_rev=`hg -R $l.beta id -i -r default`
    hg -R $l pull $l.beta
    hg -R $l up -C default
    heads=`hg -R $l heads --template '{rev}\n' default|wc -l`
    if [ "x$heads" != "x1" ]; then
        hg -R $l up -C -r $beta_rev
        HGMERGE=true hg -R $l merge -r $release_rev
        hg -R $l revert -a -y --no-backup -r $beta_rev
        hg -R $l commit -m "Merge from mozilla-beta" -u "$HG_USER"
        hg -R $l commit -u $HG_USER -m "Merge from mozilla-beta. CLOSED TREE a=release"
    fi
    hg -R $l diff -r $beta_rev -r default
    hg -R $l push -f -e "ssh -l ffxbld -i ~/.ssh/ffxbld_dsa" ssh://$HG_HOST/$release_repo_path/$l
done
</pre>

=== Tag ===
Code used:
* [http://hg.mozilla.org/build/tools/file/tip/scripts/release/tagging.sh tagging.sh]
* [http://hg.mozilla.org/build/tools/file/tip/scripts/release/tag-release.py tag-release.py]
* [http://hg.mozilla.org/build/tools/file/tip/scripts/release/version-bump.pl version-bump.pl]

Deliverables:
* Release branch and tags on source and l10n repositories

This steps creates an in-repo branch and tags in both source and locale repositories. For example, if you started Release Automation for Firefox 3.1b3 on January 17th, 2009 (building out of the mozilla-1.9.1 repository) you would end with the following in mozilla-1.9.1 and all appropriate l10n repositories:
* GECKO191_20090117_RELBRANCH named branch.
* FIREFOX_3_1b3_BUILD1 tag
* FIREFOX_3_1b3_RELEASE tag

For source repositories, after creating the branch Release Automation will perform a version bump to ensure that the release will be versioned correctly. After commiting this, it will be tagged with both of the FIREFOX_* tags and pushed back from whence it came.

For l10n repositories the exact revisions give in l10n-changesets are used as the branch point *and* tagged with the FIREFOX_* tags. They do not require a version bump.

For combined or Fennec-only releases the "tag" builder starts tagging in parallel and creates the following branches/tags in mozilla and l10n repositories:
* Release branch with MOBILE prefix
* FENNEC_${VERSION}_BUILDn tag
* FENNEC_${VERSION}_RELEASE tag

=== Source / XULRunner / Fennec Source ===
Code:
* 'SingleSourceFactory' from [http://hg.mozilla.org/build/buildbotcustom/file/tip/process/factory.py process/factory.py]
Deliverables:
* Source bundle and tarball uploaded to the staging server.

Creates a ready-to-be-built tarball and ready-to-be-used Mercurial bundle of the sourceRepo. These files will be uploaded to the candidates directory in the 'source' subdirectory.

=== Build ===
Code:
* 'ReleaseBuildFactory' from [http://hg.mozilla.org/build/buildbotcustom/file/tip/process/factory.py process/factory.py]

Deliverables:
* Desktop (Firefox)
** en-US build for each platform (tarball for Linux and Linux64, dmg for Mac, exe and zip for Windows) uploaded to the staging server
** en-US complete MAR files for each platform uploaded to the staging server
** *_info.txt files containing BuildID uploaded to the staging server
** .checksums files containing sha512 sums of all files uploaded
** debugging symbols uploaded to the symbol server
* Mobile (Fennec)
** en-US and multi APK and DEB for Android and Maemo5 platforms uploaded to the staging server
** en-US build for each mobile desktop platform (tarball for Linux, dmg for Mac, exe and zip for Windows) uploaded to the staging server
** *_info.txt files containing BuildID uploaded to the staging server
** .checksums files containing sha512 sums of all files uploaded
** debugging symbols uploaded to the symbol server

This step happens per item in enUSPlatforms and follows the nightly build process very closely. Notable differences are:
* Updates to the _RELEASE tag prior to building
* Does not run codesighs
* Does not generate update snippets
* Uses a different mozconfig

=== XULRunner Build ===
Code:
* 'XulrunnerReleaseBuildFactory' from [http://hg.mozilla.org/build/buildbotcustom/file/tip/process/factory.py process/factory.py]

Deliverables:
* XULRunner runtime (tarball for Linux, dmg for Mac, zip for Windows) and SDK per platform, uploaded to the staging server

This step happens per item in xulrunnerPlatforms and follows the nightly XULRunner build process almost identically. Notable differences are:
* Updates to the _RELEASE tag prior to building
* Uses a different mozconfig

=== L10n Repack ===
Code:
* [http://hg.mozilla.org/build/tools/file/tip/scripts/l10n/release_repacks.sh release_repacks.sh]
* [http://hg.mozilla.org/build/tools/file/tip/scripts/l10n/standalone_repacks.sh standalone_repacks.sh]
* [http://hg.mozilla.org/build/tools/file/tip/scripts/l10n/create-release-repacks.py create-release-repacks.py]
* [http://hg.mozilla.org/build/tools/file/tip/scripts/l10n/release_mobile_repacks.sh release_mobile_repacks.sh
* [http://hg.mozilla.org/build/tools/file/tip/scripts/l10n/release_mobile_repacks.py release_mobile_repacks.py]

Deliverables:
* A build for each platform/locale combination (except Android and Maemo5)
* A complete MAR file for each platform/locale combination (except Fennec)
* A langpack (.xpi) file for each platform/locale combination (except Android and Maemo5)
* .checksums file containing sha512 sums for all uploaded files

Repacks are done across a set of Builders per platform. The number of Builders per platform is controlled by the 'l10nChunks' release config variable, with a default set as DEFAULT_L10N_CHUNKS in [http://hg.mozilla.org/build/buildbotcustom/file/tip/process/release.py process/release.py]. Each Builder performs one Build; each Build repacks (totalLocales / l10nChunks) locales.

==== Chunked repacks ====
If the job fails before any locale has been processed you can just hit "rebuild".

==== Standalone Repack Builders ====
We sometimes run into the case where individual locales need to be re-spun. The easiest way to do this without re-spinning the entire chunk that failed is to use the standalone repack builder through the 'force build' form. You will need to enter the following fields and properties for the builder to succeed:

* branch
* revision (you should use the _RELEASE tag here)
* properties:
** locale (A colon separated list of locales to respin
** release_config (usually mozilla/release-mozilla-firefox-<branch name>.py)
** script_repo_revision (you should also use the _RELEASE tag here)

==== Doing an en-US only release build ====
If you want to use the autosign abilities of the automation, and are doing an en-US only build (eg. [https://wiki.mozilla.org/Releases/Firefox_5.0b1/BuildNotes 5.0b1]) then shipped-locales should be updated prior to the release to accurately reflect what is being shipped so that 'make autosign' doesn't bail on the repacks. Otherwise, manually sign your build.

=== Partner Repack ===
Code:
* 'PartnerRepackFactory' from [http://hg.mozilla.org/build/buildbotcustom/file/tip/process/factory.py process/factory.py]
* [http://hg.mozilla.org/build/partner-repacks/file/tip/scripts/partner-repacks.py partner-repacks.py]

Deliverables:
* A full set of unsigned partner repacks, uploaded to the staging server

This step is only done as part of releases on the latest stable branch. It uses unsigned en-US and l10n builds to create partner repacks.

=== Signing ===
==== Firefox ====
Code used:
* [http://hg.mozilla.org/build/tools/file/tip/release/signing/Makefile Signing Makefile]
* [http://hg.mozilla.org/build/tools/file/tip/release/signing/sign-release.py sign-release.py]
* [http://hg.mozilla.org/build/tools/file/tip/release/signing/download_builds.py download_builds.py]
* [http://hg.mozilla.org/build/tools/file/tip/release/signing/signing.py signing.py]
* [http://hg.mozilla.org/build/tools/file/tip/release/signing/verify-signature.py verify-signature.py]
* sign-release (no link available)
* sign-files (no link available)
* verify-signatures (no link available)
* checksum-files (no link available)

Deliverables:
* Signed Windows builds (internal binaries and installers), including partner repacks (where applicable)
* Detached GPG signatures for all builds and source packages
* MD5SUMS and SHA1SUMS files containing checksums for all builds, source packages, langpacks, and some MARs

This is a semi-automated process. The instructions for initiating and finishing the process are documented on [https://intranet.mozilla.org/Build:CombinedSigning the Combined Signing page].

The final step of signing is uploading the signing log. An FtpPoller watches for it to appear on the ftp server and triggers the appropriate builders when it appears.

==== Android ====
Code used:
* ~/sign_android_0.8.sh script on the keysigning machine

Deliverables:
* Signed Android APK files for en-US and multi

This is a semi-automated process. The instructions for initiating and finishing the process are documented on [https://intranet.mozilla.org/Build:MobileSigning the Mobile Signing page].

=== L10n Verify (Firefox only) ===
Code:
* 'L10nVerifyFactory' from [http://hg.mozilla.org/build/buildbotcustom/file/tip/process/factory.py process/factory.py]
* 'L10nVerifyMetaDiff' from [http://hg.mozilla.org/build/buildbotcustom/file/tip/steps/release.py steps/release.py]
* [http://hg.mozilla.org/build/tools/file/tip/release/l10n/verify_l10n.sh verify_l10n.sh]

Deliverables:
* l10n verify logs, per platform

As the name implies this is strictly a verification step. For both the current and previous release it compares the strings in the localized build with the strings in the en-US build. This diff is dumped to a file. Once this is done for every locale for the current and previous release a metadiff of $locale.diff for the current and previous release is done. If that diff has no output, no strings have changed. 

==== Log Analysis ====
There are a few common things to watch for:
* In the "compare current/previous"
** Anything about being unable to unpack or mount a package
*** Usually indicates a corrupt package
*** On Mac, can mean that '7z' is not installed or not in PATH.
* In the "Only in..." log:
** Locale(s) only in the previous release:
*** Locale may have failed to build (make sure the locale exists in the candidates directory)
*** Locale may have been removed in this release (compare shipped-locales from previous and current release)
** Locale(s) only in the current release:
*** Locale newly added in this release (compare shipped-locales from previous and current release)

The metadiff is a bit trickier and less clean cut. In a typical stable release (eg. 3.1.3) there are no string changes, the l10n metadiff log should be empty. During beta cycles there are lots of string changes, and therefore lots of output in the metadiff log. Generally, no analysis is done on the metadiff for Betas, because of the high rate of change.

=== Update/snippet Generation (Firefox only) ===
Code used:
* 'ReleaseUpdatesFactory from [http://hg.mozilla.org/build/buildbotcustom/file/tip/process/factory.py process/factory.py]
* [http://mxr.mozilla.org/mozilla/source/tools/patcher/patcher2.pl patcher2.pl and associated libraries]
* [http://hg.mozilla.org/build/tools/file/tip/release/patcher-config-bump.pl patcher-config-bump.pl]
* [http://hg.mozilla.org/build/tools/file/tip/release/update-verify-bump.pl update-verify-bump.pl]
* [http://hg.mozilla.org/build/tools/file/tip/release/generate-candidate-build-updates.py generate-candidate-build-updates.py]
* [http://hg.mozilla.org/build/tools/file/tip/release/compare-channel-snippets.sh compare-channel-snippets.sh]

Deliverables:
* Updated patcher config file, checked into CVS
* Partial mar file per-platform per-locale
* Update snippets on test & live channels
* Snippet comparison log

After checking out the necessary modules and tools this builder goes to work doing the following:
* Bumping & checking in a patcher config file
* Cloning sourceRepo & building update-packaging tools
* Creating & uploading partial MARs
* Creating & uploading update snippets
* When applicable, creating snippets for previous builds of the release
* Pushing test snippets live
* When applicable, doing a comparison of live channel snippets vs. releasetest

This step is generally pretty failsafe but in the event that it needs to be run a second time or restarted at later step it's necessary to comment out the patcher config bump step (see [[#Restarting_a_builder_from_a_certain_point | below]] for details on how). Until <strike>{{bug|466999}} FIXED</strike> is resolved the patcher config file can only be bumped once.

Note that both the sourceRepo and the tools repo will be updated to the patcherToolsTag. If the update-packaging tools or patcher config script has been updated since the last release it is a good idea (and sometimes necessary) to create a new UPDATE_PACKAGING_R# tag before starting the release. [[ReleaseEngineering/PatcherTags]] documents the tag history, please keep it up to date.

==== Snippet Comparison Log Analysis ====
Once updates is completed there will be a log from the snippet comparison step. This step compares the contents of the releasetest channel snippets to the contents of the live channel (beta or release) snippets. There's a few things to note about this log:
* It will usually contain warnings about missing snippets for alphas and betas. These are expected and ignorable.
* When there are warnings about other snippets missing, a 'Warnings' log will be created and the step will be orange. These must be investigated and explained/fixed before any live snippets are pushed.
* If any snippets differ in contents between the test and live channels a 'Diffs' log will be created and the step will be orange. The Diffs log will list the files which differ; the full diffs can be viewed in the stdio log. Any differences must be investigated and explained/fixed before any live snippets are pushed.

=== Update Verify (Firefox only) ===
Code used:
* [http://hg.mozilla.org/build/tools/file/tip/release/updates/verify.sh release/updates/verify.sh]

Deliverables:
* Update verify logs per platform. Details on them are below.

Update verify, in its current form, verifies a few different things:
* That the "updater" binary works
* That all MAR files can be applied, and result in a functionally equivalent build to the installer.
* That all previous versions (across all locales), have snippets available

It does this in two different types of tests:
* Full update verify unpacks a previous version's installer, pings AUS, downloads the MAR it finds, applies it, and diffs the result against the current version's installer. Differences between the two are reported in the log.
** Run for all locales in the oldVersion
** Run for select locales ("de", "en-US", "ru" at the time of writing) for all other previous versions
** Completes and partials (if present) are both tested
* Quick update verify does a subset of the above: it pings AUS, downloads the MAR it finds, and ensures that it is non-zero in size
** Run against previous versions for all locales not tested as part of the full update verify.

Update verify is split into multiple chunks per platform (10, by default). The full and quick checks are spaced evenly throughout the chunks giving each one roughly equivalent run time.

==== Log Analysis ====
This step is pretty good at reporting errors. Most errors will turn the step red, and be pretty clearly visible in the log. Nonetheless it's a good idea to have a scan through the log for anything abnormal (searching for FAIL and WARN are helpful here).

Here's some common failures:
* "Binary files source/bin/freebl3.chk and target/bin/freebl3.chk differ" (or softokn3.chk) on win32 in the complete MAR
** This is normal in the win32 logs and is caused by {{bug|404340}}. Once this bug is fixed it should go away.
* "CRC check failed" and/or "LoadSourceFile failed" on a partial MAR
** This typically happens when the previous release's complete MAR differs from the complete package. Because partial MARs are generated by diffing the previous complete MAR with the current complete MAR the partial will fail the CRC check when this happens. There's nothing that can be done about this, because the problem is with an already-release-build. The correct thing to do here is inform release-drivers that there will not be partials available on this platform, trim them out of the candidates directory, and remove the partial snippets.
* "FAIL: partial from xxxxx wrong size", "FAIL: actual size: 0"
** This is usually caused by missing MARs. You should first check to see if the MAR in question exists in the candidates directory. If it doesn't, go back and look at the logs to see if there was a problem generating or uploading it. If it does exist in the candidates directory make sure the permissions are set properly (ffxbld:firefox, 664).
* "FAIL: Could not download source xxx"
** Usually indicates missing packages or bad permissions in the candidates directory. Take the same action as above.
* "FAIL: no partial update found for xxx" or "FAIL: no complete update found for xxx"
** Check the logs from the previous step to make sure 'pushsnip' was run successfully. This error usually indicates that snippets weren't pushed.

=== Major Update (Firefox only) ===
Code:
* [http://hg.mozilla.org/build/tools/file/tip/release/patcher-config-creator.pl patcher-config-creator.pl]
* [http://hg.mozilla.org/build/tools/file/tip/release/update-verify-bump.pl update-verify-bump.pl]
* 'MajorUpdateFactory' from [http://hg.mozilla.org/build/buildbotcustom/file/tip/process/factory.py process/factory.py]
* [http://mxr.mozilla.org/mozilla/source/tools/patcher/ patcher2.pl and libraries]

As part of any release that is on a less than current branch we generally create and push a Major Update as part of it. For example, when we did a 3.5.8 release we generated a major update for 3.5.8 -> 3.6 alongside it. Note that major updates happen as part of the "from" release, not the "to" release.

Major update generation is not kicked off automatically because it depends on both the "to" and "from" releases having had their builds and repacks signed. Once they are at that point "Force Build" can be used to initiate the Major Update generation.

If <b>either</b> of the "from" or "to" releases respin at any point you must regenerate the major update. If the "to" release respins, majorUpdateBuildNumber needs to be updated.

At the end of the factory snippets will be uploaded to the AUS server and the test snippets will be pushed live. After that, the major update verify builders will be triggered.

=== Major Update Verify (Firefox only) ===
Everything in the previous "update verify" section applies here.

Generally, major updates generate more harmless warnings than other releases. Analysis on these is done when testing a new major update and noted on release notes. When verifying a major update it's always helpful to look at the previous one's notes to find a list of the harmless warnings. If you find warnings that are NOT in the previous major update's notes they must be investigated.

It's rare to see major update verify builders go green because of all of the harmless warnings.

=== Pre Push to Mirrors (Firefox only) ===

Code: [http://hg.mozilla.org/build/tools/file/tip/scripts/release/push-to-mirrors.py scripts/release/push-to-mirrors.py]

==== Check Permissions ====
This builder:
# Checks if the FTP directories and files have proper permissions and modes
# Runs rsync in dry-run mode

This builder is triggered automatically after the updates builder.

This builder should be run second time before pushing file to mirrors by hitting "Rebuild" button or forcing the builder with 2 properties set:
* '''Property 1 name''': script_repo_revision: use FIREFOX_*_RELEASE tag here
* '''Property 2 name''' -> release_config: mozilla/release-firefox-mozilla-2.0.py

No reconfig needed for this builder in manual mode. Automatic mode requires reconfig only on version bump.

==== Antivirus check ====

This builder:
# Runs antivirus
# Runs rsync in dry-run mode

This builder is triggered automatically after the updates builder.

This builder should be run second time before pushing files to mirrors by hitting "Rebuild" button or forcing the builder with 2 properties set:
* '''Property 1 name''': script_repo_revision use FIREFOX_*_RELEASE tag here
* '''Property 2 name''' -> release_config: mozilla/release-firefox-mozilla-2.0.py

No reconfig needed for this builder in manual mode. Automatic mode requires reconfig only on version bump.

=== Push to Mirrors (Firefox only) ===
Code: [http://hg.mozilla.org/build/tools/file/tip/scripts/release/push-to-mirrors.py scripts/release/push-to-mirrors.py]

This builder:
# Runs rsync and pushes files to the release directory
# Triggers mirror uptake monitoring poller.

To trigger this builder you need to specify some build properties:
* '''Property 1 name''': script_repo_revision use FIREFOX_*_RELEASE tag here
* '''Property 2 name''' -> release_config: mozilla/release-firefox-mozilla-2.0.py
Press 'Force Build' in the Buildbot waterfall.

No reconfig needed for this builder.

=== Final Verification (Firefox only) ===
Code:
* 'ReleaseFinalVerification' from [http://hg.mozilla.org/build/buildbotcustom/file/tip/process/factory.py process/factory.py]
* [http://hg.mozilla.org/build/tools/file/tip/release/final-verification.sh final-verification.sh]

Once mirror uptake has reached 50 or 60% this builder can be started with 'Force Build' from the Buildbot waterfall.

This is a simple test which is run after mirrors are propagated to ensure there's no (major) problems with them. It does the same thing as the second part of Update Verify: for every locale/platform/version combination it checks for an update snippet, and then performs an HTTP HEAD request on the MAR the snippet points to.

The step will turn red if there are http codes other than 200 or 302, or if FAIL is found in the log. If it does this, you should have a look in the log, and find out which mirror is failing. It's a good idea to keep an eye on that mirror for awhile to make sure it finishes getting all of the files.

Searching the log for errors is made easier by downloading it and running the following locally:
 grep HTTP/1 text | grep -v 302 | grep -v 200
 grep FAIL text

=== Bouncer Submitter (Firefox only) ===
Code:
* 'TuxedoEntrySubmitterFactory' from [http://hg.mozilla.org/build/buildbotcustom/file/tip/process/factory.py process/factory.py]
* [http://hg.mozilla.org/build/tools/file/tip/release/tuxedo-add.py tuxedo-add.py]
Configuration:
* [http://hg.mozilla.org/build/tools/file/tip/release/firefox-tuxedo.ini firefox-tuxedo.ini] for stable versions, and [http://hg.mozilla.org/build/tools/file/tip/release/firefox-devpreview-tuxedo.ini firefox-devpreview-tuxedo.ini] for Alpha releases.
* BuildSlaves.py file should contain the following variables:
** tuxedoUsername: Tuxedo username with appropriate permissions for adding new entries via API
** tuxedoPassword: password for this user

This builder can be started any time with 'Force Build' from the Buildbot waterfall.

This step adds adds bouncer entries for installers, complete and partial updates for all platforms listed in enUSPlatforms and l10nPlatforms and locales specified in shipped-locales (en-US is being added in any case).

The step will turn red if tuxedo-add.py exits non zero. There are 2 major error sources:
* tuxedoUsername has not enough privileges to add entries
* Tuxedo is not aware of new locales added in the current version. Usually the log contains "HTTPError: HTTP Error 400: BAD REQUEST". See [[Releases/Firefox_4.0b7/BuildNotes#Update_Bouncer|this link]] for an example workflow on how to resolve this problem.

== Common Problems & Resolutions ==
=== Restarting the automation from a specific point ===
The automation has the ability to easily skip tagging, source generation, or en-US builds in the event that you want to restart from source, en-US, or l10n repacks. To make use of this, you need to add skip_$builder flags to the release config. For example, to restart from l10n repacks you will need to add the following lines to the release config:
 releaseConfig['skip_tag'] = 1
 releaseConfig['skip_source'] = 1
 releaseConfig['skip_build'] = 1

Only these three steps (and repo_setup, if you're in staging) are skippable. If you need to restart in later parts of the automation, "Force Build" is the supported method of doing so.

=== Restarting failed builders without patching the config ===
It's often easier to use Force Build or buildbot sendchange to restart specific things than it is to patch the configs, check in, update the master, etc. If you need to restart tag, source, an en-US build, l10nverify, updates, or update_verify you can use "Force Build" from the Buildbot waterfall.

If you need to restart a single l10n build (that is, one locale) on a single platform you may also use "Force Build", but be sure to use en_revision, l10n_revision, and locale as properties. The revision properties should be set to _RELEASE tags, and the locale should be obvious.

If you need to restart many or all locales you'll have to fill out the web interface multiple times, until {{bug|518589}} is resolved.

=== Restarting a builder from a certain point ===
In some cases only part of a Builder will successfully complete. For example, if the 'updates' builder was able to build partial MARs successfully but failed to upload them you will want to figure out what the problem is, and then restart it from the upload step. To do that, open up /tools/buildbotcustom/buildbotcustom/process/factory.py and comment out all steps before uploading. You should also find out which slave it was started on, and update the 'slavenames' for 'updates' in release_master.py to ensure the job gets started on the same slave.

Depending on how much of a builder is left to run it can be faster and/or easier to simply run the steps manually and continue on. For example, if backupsnip or pushsnip in 'updates' failed, it's probably easier to run those steps manually rather than go to the trouble of starting another job.

In other cases, it may make sense just to start from the beginning. Eg, if the build tools repository fails to clone, it's generally easier just to start from scratch.

Use your judgment here.

=== Tagging failed out part way through ===
If this happens, you'll need to continue it on but get it to skip locales which it has already tagged. To do so, delete any locales which have been tagged from l10n-changesets and reconfig. If the source repository has already been tagged you should pass 'l10n_repositories' to ReleaseTaggingFactory instead of 'repositories. Note that the rest of the release automation uses shipped-locales so removing things from l10n-changesets doesn't cause locales not to build.

=== Re-spinning a single locale ===
We sometimes run into the case where individual locales need to be re-spun. Some reasons this might happen include network timeouts or build slave failures. When this happens, you can follow the following steps to recover and re-spin the missing locale(s):
# Manually re-tag the locale's repository (if necessary). You can find the appropriate revision to use for tagging from the shipped-locales file, or from the [https://l10n-stage-sj.mozilla.org/shipping/milestones l10n shipping dashboard]
# Delete the current build of the locale and cleanup l10n build dirs on build slaves.
# Manually force the repack on each of the "$platform_standalone_repack" builders. See the [[#Standalone_Repack_Builders|Standalone_Repack_Builders]] section for details.
# Manually sign it and update the *SUMS files
#* You need to download the new locale builds to the signing machine, but you also need the SUM files, en-US Windows build (used for caching) and the zh-TW builds (monitoring tools check for this locale to know when the directory has changed). 
#* Run <code>sign-files</code> on the new builds.
#* Manually update the SUMS files with new md5/sha1 sums.
#* Remove the .asc file for the en-US Windows build
#* Push the signed builds back to stage.
#* The [[Releases/Firefox_3.6b4/BuildNotes#signing|Build Notes for 3.6b4]] show an example of how this is done.
# Manually create a partial MAR for the locale
#* use an appropriate patcher_config file for your release.
#* on a linux slave (preferably a fast one), download the builds with <code>patcher2.pl</code>
#* use <code>patcher2.pl</code> to create the update MAR files and snippets.
#* ensure file (755) and directory (644) modes are correct for your created files.
#* transfer the MAR files to stage
#* transfer the update snippets to the aus2 server(s) <- there may be more than one
#** it is good practice to use a new directory name on the aus2 server to mark the new snippets as part of a distinct respin, e.g. <code>20091125-Firefox-3.6b4-fr-respin-test</code>. Please also add that new directory name to the list of directories to be run through backupsnip/pushnip in the build notes.
#* The [[Releases/Firefox_3.6b4/BuildNotes#updates|Build Notes for 3.6b4]] show an example of how this is done.
# Re-run the update verify builder from the waterfall.

== Staging Specific Notes ==
Release automation in staging is mostly the same as in production, but does have a few differences you should known about:
* All uploading (builds, snippets, symbols) is done to dev-stage01.build.mozilla.org
* It can take a few tries to get repo_setup to run properly. This is because hgweb sometimes returns a 500 (internal server error) when we query about a locale. The best solution is just to start the automation from scratch until it works, to make sure you get a clean run. If this is too frustrating for you, you can manually clone the repositories you care about and start automation from tag (see [[#Restarting_the_automation_from_a_specific_point | Restarting from a specific point]]).
* Staging doesn't have a lot of slaves, and you may need to go around and stop running builds to get your automation run to happen in a reasonable period of time.

=== Staging specific preparation ===
==== Download the previous release ====
Because we point the staging releases at dev-stage01.build.mozilla.org the previous release must be downloaded to it. This is done by the "release_downloader" builder which is fired by a sendchange sent by release_sanity.py. It automatically removes any candidates or relase dir on dev-stage01 which already exists.

The builder could be disabled by setting releaseConfig['skip_release_download'] = True

===== Doing it by hand =====
If you need to download the previous release by hand for some reason you can use the following shell script to do so. Note that the variables are for the *previous* release, not the one you will be running:
 # ffxbld @ dev-stage01
 export VERSION=3.5.3
 export BUILD=1
 export PRODUCT=firefox
 cd /home/ftp/pub/$PRODUCT/nightly
 mkdir -p $VERSION-candidates/build$BUILD
 cd $VERSION-candidates/build$BUILD
 wget -r -np -nH --cut-dirs=6 -R index.html* -R *unsigned* http://stage.mozilla.org/pub/mozilla.org/$PRODUCT/nightly/$VERSION-candidates/build$BUILD/
 cd /home/ftp/pub/$PRODUCT/releases
 ln -s /home/ftp/pub/$PRODUCT/nightly/$VERSION-candidates/build$BUILD $VERSION

If you're doing a testrun with a limited number of locales you may delete any locales you don't care about after the above script finishes. (or you can add "-R *locale*", etc for each unwanted locale)

==== Buildbot configs ====
When working in staging you'll need to swap out the buildbot-configs, build/tools, buildbotcustom, and compare-locales repos for either stage-ffxbld ones, or your own. Here's an example of the changes you'll need:
<pre>
diff --git a/mozilla/config.py b/mozilla/config.py
--- a/mozilla/config.py
+++ b/mozilla/config.py
@@ -43,1 +43,1 @@ GLOBAL_VARS = {
-    'compare_locales_repo_path': 'build/compare-locales',
+    'compare_locales_repo_path': 'users/bhearsum_mozilla.com/compare-locales',
diff --git a/mozilla/staging_config.py b/mozilla/staging_config.py
--- a/mozilla/staging_config.py
+++ b/mozilla/staging_config.py
@@ -37,2 +37,2 @@ GLOBAL_VARS = {
-    'config_repo_path': 'build/buildbot-configs',
-    'buildbotcustom_repo_path': 'build/buildbotcustom',
+    'config_repo_path': 'users/bhearsum_mozilla.com/buildbot-configs',
+    'buildbotcustom_repo_path': 'users/bhearsum_mozilla.com/buildbotcustom',
diff --git a/mozilla/staging_release-firefox-mozilla-1.9.2.py b/mozilla/staging_release-firefox-mozilla-1.9.2.py
--- a/mozilla/staging_release-firefox-mozilla-1.9.2.py
+++ b/mozilla/staging_release-firefox-mozilla-1.9.2.py
@@ -106,1 +106,1 @@ releaseConfig['doPartnerRepacks']    = F
-releaseConfig['partnersRepoPath']    = 'users/stage-ffxbld/partner-repacks'
+releaseConfig['partnersRepoPath']    = 'users/armenzg_mozilla.com/partner-repacks'
@@ -132,1 +132,1 @@ releaseConfig['enable_repo_setup'] = Tru
-releaseConfig['build_tools_repo_path'] = "users/stage-ffxbld/tools"
+releaseConfig['build_tools_repo_path'] = "users/asasaki_mozilla.com/tools"
</pre>

As with production releases, you must tag these repositories with the _RELEASE tag prior to starting the release.


It is possible to use your own HG repository (long term release tests, parallel run, etc). Instead of running repo_setup builder, you can use the following script (run it on your laptop, using LDAP credentials):
<pre>
#!/bin/bash

for repo in compare-locales tools buildbotcustom buildbot buildbot-configs partner-repacks; do
    echo "deleting $repo"
    ssh hg.mozilla.org edit $repo delete YES
    echo "cloning $repo"
    ssh hg.mozilla.org clone $repo build/$repo
done
 
for l in `wget -q -O- http://hg.mozilla.org/mozilla-central/raw-file/tip/browser/locales/shipped-locales |grep -v en-US | awk '{print $1}'`; do
    echo "deleting locale $l"
    ssh hg.mozilla.org edit $l delete YES
    echo "cloning locale $l"
    ssh hg.mozilla.org clone $l l10n-central/$l
done

ssh hg.mozilla.org edit mozilla-central delete YES
ssh hg.mozilla.org clone mozilla-central mozilla-central
</pre>

=== Doing a test run with a limited number of locales ===
To run a test with a limited number of locales, do the following:
# Modify l10n-changesets to include whichever locales you want.
# Reconfig Buildbot and start the automation as normal
# Once the tag builder is finished and before the end of the first en-US build, clone the staging source repository (eg, http://hg.mozilla.org/users/stage-ffxbld/mozilla-1.9.1) do the following:
 hg up -C GECKO191_20090115_RELBRANCH # replace the relbranch appropriately, of course
 # modify browser/locales/shipped-locales to include the same locales as l10n-changesets
 hg commit -m "Reduce number of locales for this test run"
 hg tag -f FIREFOX_3_1b3_RELEASE # using the right version number in the tag
 hg push ssh://hg.mozilla.org/users/stage-ffxbld/mozilla-1.9.1

If you don't want to clone the full repo you can do this on the slave that did the tagging.

=== How to sign in staging ===
==== Signing with the staging keys ====
When you're specifically testing something related to signing or doing a full end to end run it's best to sign the builds on the staging signing server. Doing so is very similar to signing production builds and fully documented [https://intranet.mozilla.org/Build:CombinedSigning#Signing_in_the_Staging_Environment in the CombinedSigning doc].

==== Faking it out ====
If you're not looking to test signing you can speed up your staging run a bit by shuffling files around so post-signing steps can find them. To do this, log onto dev-stage01.build.mozilla.org and do the following:
 # ffxbld@dev-stage01
 VERSION=3.5rc1
 BUILD=1
 cd /home/ftp/pub/firefox/nightly/${VERSION}-candidates/build${BUILD}
 mkdir win32 update/win32
 rsync -av --exclude=*.zip unsigned/win32/ win32/
 rsync -av unsigned/update/win32/ update/win32/
 rsync -av unsigned/win32_info.txt .
 echo "faked" > win32_signing_build${BUILD}.log

We purposely make copies here rather than symlinking for a couple of reasons: L10n verify scripts barf when they get zip files (hence the --exclude above), 'updates' factory will blow away complete MARs upon upload if update/win32 is a symlink. The echo creates the log the automation looks for, in order to continue to l10n verify and updates.

=== Creating a CVS mirror for patcher and configs ===
If you need a new or modified patcher config, which shouldn't be checked into production CVS, then modify the following method:
 WHO=yournamehere
 # cltbld@dev-stage01
 cd /builds/cvsmirrors
 mkdir -p ${WHO}/cvsroot.clean/mozilla/tools/
 rsync -av --exclude=CVSROOT/config --exclude=CVSROOT/loginfo cvs-mirror.mozilla.org::mozilla/CVSROOT /builds/cvsmirrors/${WHO}/cvsroot.clean/
 rsync -av cvs-mirror.mozilla.org::mozilla/mozilla/tools/patcher-configs /builds/cvsmirrors/${WHO}/cvsroot.clean/mozilla/tools/
 rsync -av cvs-mirror.mozilla.org::mozilla/mozilla/tools/patcher /builds/cvsmirrors/${WHO}/cvsroot.clean/mozilla/tools/
 rsync -av cvs-mirror.mozilla.org::mozilla/mozilla/tools/release /builds/cvsmirrors/${WHO}/cvsroot.clean/mozilla/tools/
 rsync -a --delete-after /builds/cvsmirrors/${WHO}/cvsroot{.clean,}/
To make changes check out using 
 cvs -d dev-stage01.build.mozilla.org:/builds/cvsmirrors/${WHO}/cvsroot co mozilla/
and specify a cvsroot of <tt>:ext:cltbld@dev-stage01.build.sjc1.mozilla.com:/builds/cvsmirrors/${WHO}/cvsroot</tt> in the config for the release automation.