Notes About Releasing

Please update the Notes Template and the Release:Primer for future releases (bug fixes, changes to automation) as needed

Bugs hit

Enter any bugs pre-existing or newly discovered and filed during the release:

bug 773876 - partner repack's download_pkg-dmg step does not retry failed wget

Build Engineers

nthomas with a start from bhearsum - Tracking bug: bug 758795

Signed-off Revision(s)

Build 1: 228ba1a111fc

L10N changesets

Instructions on how to get them

Build data

Firefox

Build #	Type	Build ID	Build machine	Time to build
1	Linux	20120713134347	inux-ix-slave39	1 hrs, 9 mins, 10 secs
	Linux64		linux64-ix-slave16	1 hrs, 4 mins, 38 secs
	Mac		bld-lion-r5-063	1 hrs, 19 mins, 34 secs
	Windows		mw32-ix-slave06	3 hrs, 41 mins, 16 secs

Fennec

Build #	Type	Build ID	Build machine	Time to build
1	android	20120713133725	linux-ix-slave06	50 mins, 41 secs
2	android	20120713170118	linux-ix-slave39	52 mins, 49 secs

Notes

Build 1

Diff mozconfigs

From a mozilla-release checkout (with 13.0.1 content), ran:

for p in linux32 linux64 macosx-universal win32; do diff -Naur {,../mozilla-beta/}browser/config/mozconfigs/$p/release; done

(note: mozilla-beta had not been updated to 15.0 code yet.)

No differences, nothing to do here.

Set reserved slaves

set reserved_slaves to 8:

cd /builds/buildbot/build1
echo 8 > master/reserved_slaves

Preparing to start Automation

set a clobber for any master, release-mozilla-release, any builder as a speed up
create a symlink to the candidates directory

# ffxbld or tbirdbld@stage
version=14.0.1
product=firefox
cd /pub/mozilla.org/$product/nightly/
mkdir ../candidates/$version-candidates
ln -s ../candidates/$version-candidates $version-candidates

moved the slavealloc lock for 6 mw32-ix-slave from bm13 to bm12, rebooted them. mw32-ix-slave04 needed TLC for fuzzing exhausting e drive (bug 771687). Plenty of linux/linux64 slaves already. Locked 8 bld-r5-lion slaves to bm12.
Shipped the l10n milestones
- Clicked "Ship" for Fennec
  - Clicked "Add"
    - repo: releases/mozilla-release
    - branch: default
    - path: mobile/android/locales/maemo-locales
  - Clicked "Ship it"
- Clicked "Ship" for Firefox
  - Clicked "Ship it"
Landed buildbot-configs patch w/ updated l10n-changesets:

wget -O- --no-check-certificate "https://bugzilla.mozilla.org/attachment.cgi?id=641885" | patch -p1
sed -i -e "s/TBD/'228ba1a111fc'/" mozilla/release-firefox-mozilla-release.py mozilla/release-fennec-mozilla-release.py
wget -O mozilla/l10n-changesets_mobile-release.json "https://l10n-stage-sj.mozilla.org/shipping/json-changesets?ms=fennec14.0.1&platforms=android&multi_android-multilocale_repo=releases%2Fmozilla-beta&multi_android-multilocale_rev=default&multi_android-multilocale_path=mobile%2Fandroid%2Flocales%2Fmaemo-locales"
wget -O mozilla/l10n-changesets_mozilla-release "https://l10n-stage-sj.mozilla.org/shipping/l10n-changesets?ms=fx14.0.1"
# Remove the locales we aren't shipping from l10n changesets
sed -i -e '/^mn.*$/d' -e '/^sw.*$/d' mozilla/l10n-changesets_mozilla-release
hg commit -m "bug 758795: tracking bug for build and release of Firefox/Fennec 14.0.1. r=aki"
hg up -r production
hg transplant default
hg tag -r production {FIREFOX,FENNEC}_14_0_1_{RELEASE,BUILD1}
hg push ssh://hg.mozilla.org/build/buildbot-configs

Tagged buildbotcustom:

hg tag -r production-0.8 {FIREFOX,FENNEC}_14_0_1_{RELEASE,BUILD1}
hg push ssh://hg.mozilla.org/build/buildbotcustom

Tagged tools:

hg tag -r default {FIREFOX,FENNEC}_14_0_1_{RELEASE,BUILD1}
hg push ssh://hg.mozilla.org/build/tools

Updated and reconfiged buildbot-master12:

cd /builds/buildbot/build1
source bin/activate
make update reconfig

Ran release sanity in dry run mode:

cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u bhearsum \
   -V 14.0.1 --branch mozilla-release --build-number 1 \
   --release-config release-firefox-mozilla-release.py \
   --release-config release-fennec-mozilla-release.py --products firefox,fennec  \
   --dryrun localhost:9001

- Complained about mn and sw being missing, because the l10n dashboard is wrong
Ran release sanity with --bypass-l10n-check to get the release going:

PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u bhearsum \
   -V 14.0.1 --branch mozilla-release --build-number 1 \
   --release-config release-firefox-mozilla-release.py \
   --release-config release-fennec-mozilla-release.py --products firefox,fennec  \
   --bypass-l10n-check localhost:9001

E-mail Metrics

Sent the following mail to metrics-alerts@mozilla.org:

Firefox,firefox,firefox,14.0.1,14.0
Fennec,firefox-mobile,fennec,14.0.1,14.0

E-mail mirrors@mozilla.org

nthomas will do this later

Edit rsync exclude files

Committed this diff, after review:

Index: rsyncd-mozilla-releases.exclude
===================================================================
--- rsyncd-mozilla-releases.exclude	(revision 43094)
+++ rsyncd-mozilla-releases.exclude	(working copy)
@@ -1,3 +1,12 @@
++ firefox
++ firefox/releases
++ firefox/releases/14.0.1
++ firefox/releases/14.0.1/*/
++ firefox/releases/14.0.1/*/*/
++ firefox/releases/14.0.1/*/*/*/
++ firefox/releases/14.0.1/**index.html
+- firefox/releases/14.0.1/**
+- firefox/releases/13.0.1-funnelcake11/update
 - firefox/releases/0.10
 - firefox/releases/0.10.1
 - firefox/releases/0.10rc

Tag

No problems

Bouncer Submitter

Bogus 13.0.2 oldVersion (see Updates for diagnosis) resulted in

renaming Firefox-14.0.1-Partial-13.0.2 to Firefox-14.0.1-Partial-13.0.1
s/13.0.2/13.0.1/ in the 8 locations for that product

Additional work still to be done for funnelcake, see notes. Yes, it's a bit strange to put 14.0.1 updates in 13.0.1 dirs, but we already have a set up to block CDN/mirrors as needed.

Source

Failed because I removed the buildbot-configs mozconfig that this step depends on. Reverted with:

hg revert -r 6507 mozilla2/linux/mozilla-release/release/mozconfig
hg commit -m "Re-add linux mozconfig so source steps will work"
hg up -r production
hg transplant -r production
hg tag -f -r production {FIREFOX,FENNEC}_14_0_1_{RELEASE,BUILD1}
hg push ssh://hg.mozilla.org/build/buildbot-configs

Used "rebuild" to restart source steps.

Failed again with a configure error. Realized this was because I didn't merge the beta mozconfig -> production. Brought forward compiler changes in http://hg.mozilla.org/build/buildbot-configs/rev/466340b368b8 to fix this. Used rebuild to try again.

Failed again with the same error. I did some debugging on the machine and found that there were some CFLAGS/CXXFLAGS I didn't bring forward and did that in http://hg.mozilla.org/build/buildbot-configs/rev/b2b6a37879f2. Used rebuild to try again. Worked this time.

Build

Firefox

No problems

Firefox repacks

No problems

Fennec

No problems

Fennec repacks

No problems

XULRunner

Source

Failed with the same issue as Firefox source builder.

Build

Mac build failed first time because of bug 753223. Rebuilt it, got success
Other platforms no problems

Push to Mirrors

No problems

Partner Repack

Firefox

Mac jobs failed when it got a 500 error trying to d/l pkg-dmg from hg.m.o, bug 773876 filed. Success on rebuild. Other platforms fine.

TODO Come back and generate funnelcake13/14 on win32/mac.

Android Signing

FENNEC ONLY RESPIN DECLARED.

Checksums

No problems, but will need to rerun for 12.0 to 14.0.1 partials.

Updates

First problem: Failed to bump the patcher config

perl ../tools/release/patcher-config-bump.pl -p firefox -r Firefox -v 14.0.1 -a 14.0.1 \
-o 13.0.2 -b 1 ....
ASSERT: BumpFilePath() - Could not bump path: 'firefox/nightly/13.0.1-candidates/build1/update/%platform%/%locale%/firefox-13.0-13.0.1.partial.mar' from '13.0.2' to '14.0.1' at ../tools/lib/perl/Release/Patcher/Config.pm line 148.

due to incorrect oldVersion in release config.

Fix:

# buildbot-configs @ ed8561aa1056
sed -e 's/13_0_2/13_0_1/' -e 's/13\.0\.2/13.0.1/' -i  mozilla/release-firefox-mozilla-release.py
hg ci -m "Bug 758795, fix up oldVersion for FF 14.0.1, r=version-numbers-are-confusing"
hg up -r production
hg transplant 1aa1d4209bb1
hg tag -f -r production {FIREFOX,FENNEC}_14_0_1_{RELEASE,BUILD1}
hg push ssh://hg.mozilla.org/build/buildbot-configs

# bm12
cd /builds/buildbot/build1
source bin/activate 
make update reconfig

Note that the tagging was wrong for Fennec, revert it

hg tag -f -r b52a7634bca669ea1205ad34ab297b845ec193c1 FENNEC_14_0_1_RELEASE
hg tag -f -r b2b6a37879f23438c3fb29124f9b3d37e61cff13 FENNEC_14_0_1_BUILD1
# after verification using hg serve
hg push ssh://hg.mozilla.org/build/buildbot-configs

Forced builder.

Second problem: Failed to upload aus2-test because in the create_snippets step we got

Partial patch info - 0 to create
Complete patch info - 0 to create
Past release patch info - 6168 to create
No known patch for locale af, 10.0 -> 14.0.1; skipping...

instead of generating snippets. This is because the checksums files have lines like this

26dee2baa2b425fb3d8057879558bf89 md5 7771770 update/win32/en-US/firefox-13.0.2-14.0.1.partial.mar

Which is the same problem with the wrong version in the config. In fact we have generated partials for 13.0.2 to 14.0.1, which won't be any use, and will need to create 13.0.1 to 14.0.1 partials.

Manually generate 13.0.1-14.0.1 partials

Working from the howto doc, but no config changes are necessary here.

## on linux-ix-slave36
cd /builds/slave/rel-m-rel-updates/build
mkdir -p logs/13.0.1-14.0.1

## build tools (9 min)
export HGROOT=http://hg.mozilla.org/releases/mozilla-release
perl patcher2.pl --build-tools-hg --tools-revision=FIREFOX_14_0_1_RELEASE \
  --app=firefox --brand=Firefox \
  --config=patcher-configs/mozRelease-branch-patcher2.cfg

## get complete mar files   (75 min, very badly needs parallelization)
perl patcher2.pl --download \
  --app=firefox --brand=Firefox \
  --config=patcher-configs/mozRelease-branch-patcher2.cfg \
  2>&1 | tee logs/13.0.1-14.0.1/download.log

## generate unsigned partial updates (100 min)
perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg \
 --app=firefox --brand=Firefox \
 --config=patcher-configs/mozRelease-branch-patcher2.cfg \
  2>&1 | tee logs/13.0.1-14.0.1/create-partials.log

## Make a backup of the new partials/snippets, destroy snippets (6 min)
rsync -a temp/firefox/13.0.1-14.0.1{,-unsigned}/
rm -rf temp/firefox/13.0.1-14.0.1/aus2*

## Sign the mar files (< 1 min)
# get a token
# look up the values for --user in the release-signing part of passwords.py, on the master
unset HISTFILE
IP=`/sbin/ifconfig eth0 | grep "inet addr" | awk -F: '{print $2}' | awk '{print $1}'`
ssh -oBatchMode=no cltbld@buildbot-master12.build.mozilla.org \
  "curl --fail -k -Fslave_ip=$IP -Fduration=3600 --user reda:cted \
   https://signing1.build.scl1.mozilla.com:9120/token" \
     > /builds/slave/rel-m-rel-updates/token
rm /builds/slave/rel-m-rel-updates/nonce

# update the channel id and version, then sign (25 min)
(for f in `find temp/firefox/13.0.1-14.0.1/ftp -type f -name *partial.mar | sort`; do
  echo Signing $f
  mozilla/obj/dist/host/bin/mar -H firefox-mozilla-release -V 14.0.1 -i $f;
  /tools/python-2.6.5/bin/python \
    /builds/slave/rel-m-rel-updates/tools/release/signing/signtool.py \
    -t /builds/slave/rel-m-rel-updates/token -n /builds/slave/rel-m-rel-updates/nonce \
    -c /builds/slave/rel-m-rel-updates/tools/release/signing/host.cert \
    -H signing1.build.scl1.mozilla.com:9120 -H signing2.build.scl1.mozilla.com:9120 -f gpg -f mar \
    "$f"
done ) 2>&1 | tee logs/13.0.1-14.0.1/sign-partials.log

## Regenerate the snippets (4 min)
perl patcher2.pl --create-patchinfo \
  --app=firefox --brand=Firefox \
  --config=patcher-configs/mozRelease-branch-patcher2.cfg \
  2>&1 | tee logs/13.0.1-14.0.1/generate_snippets.log

# Snippet verification
cd temp/firefox/13.0.1-14.0.1
bash /builds/slave/rel-m-rel-updates/tools/release/compare-channel-snippets.sh \
  aus2 release aus2.test  releasetest

# Tidy and upload mar files (6 min)
find ftp/firefox/nightly/14.0.1-candidates/build1 -type f -exec chmod 644 {} \; 
find ftp/firefox/nightly/14.0.1-candidates/build1 -type d -exec chmod 755 {} \;
# $BUILD is the build number, eg 1, 2, ...
rsync -nav -e 'ssh -oIdentityFile=~/.ssh/ffxbld_dsa' '--exclude=*complete.mar' \
  ftp/firefox/nightly/14.0.1-candidates/build1/update \
  ffxbld@stage.mozilla.org:/pub/mozilla.org/firefox/nightly/14.0.1-candidates/build1/
# then repeat without the -n

# Upload snippets (2 min, NB: no  --exclude=complete.txt)
rsync -av -e 'ssh -oIdentityFile=~/.ssh/auspush' \
 aus2.test/ \
 ffxbld@aus3-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-14.0.1-build1-test
rsync -av -e 'ssh -oIdentityFile=~/.ssh/auspush' \
 aus2/ \
 ffxbld@aus3-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-14.0.1-build1

Defer backupsnip until after other changes.

Adjust What's New in snippets

From bug 767207

locales="en-US en-GB fr it de es-ES es-AR es-CL es-MX ja ja-JP-mac ko ru"

# backup auto-generated snippets
cd /opt/aus2/snippets/staging
rsync -a Firefox-14.0.1-build1{,-withoutWN}/
rsync -a Firefox-14.0.1-build1-test{,-withoutWN}/

for l in $locales; do
  for f in `find Firefox-14.0.1-build1{,-test}/*/*/*/*/$l -type f`; do
    echo tweaking $f
    sed -i '/^actions.*$/d' $f
  done
done

Did some quick checks using diff, looks OK.

Generate 12.0 to 14.0.1 partial update for windows

## on linux-ix-slave36
cd /builds/slave/rel-m-rel-updates/build
mkdir -p logs/12.0-14.0.1
curl -sL https://bugzilla.mozilla.org/attachment.cgi?id=642484 \
  > patcher-configs/mozRelease-branch-patcher2-758795.cfg

# save the previous partial working space, but duplicate 14.0.1
# win32 completes as a speed up (2 min)
mv temp/ temp-13.0.1-14.0.1-partial/
mkdir -p temp/firefox/14.0.1/ftp
cp -pv temp-13.0.1-14.0.1-partial/firefox/14.0.1/ftp/*win32* temp/firefox/14.0.1/ftp/

## build tools - skipped as we had this from 12.0.1 - 13.0.1

## Get the complete mar files (8 min)
perl patcher2.pl --download \
  --app=firefox --brand=Firefox \
  --config=patcher-configs/mozRelease-branch-patcher2-758795.cfg

## Generate unsigned partial updates (28 min)
perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg \
 --app=firefox --brand=Firefox \
 --config=patcher-configs/mozRelease-branch-patcher2-758795.cfg \
  2>&1 | tee logs/12.0-14.0.1/create-partials.log

## Make a backup of the new partials/snippets, destroy snippets (< 1 min)
rsync -a temp/firefox/12.0-14.0.1{,-unsigned}/
rm -rf temp/firefox/12.0-14.0.1/aus2*

## Sign the mar files (< 1 min)
# get a token
# look up the values for --user in the release-signing part of passwords.py, on the master
unset HISTFILE
IP=`/sbin/ifconfig eth0 | grep "inet addr" | awk -F: '{print $2}' | awk '{print $1}'`
ssh -oBatchMode=no cltbld@buildbot-master12.build.mozilla.org \
  "curl --fail -k -Fslave_ip=$IP -Fduration=3600 --user reda:cted \
   https://signing1.build.scl1.mozilla.com:9120/token" \
     > /builds/slave/rel-m-rel-updates/token
rm /builds/slave/rel-m-rel-updates/nonce

# update the channel id and version, then sign (8 min)
(for f in `find temp/firefox/12.0-14.0.1/ftp -type f -name *partial.mar | sort`; do
  echo Signing $f
  mozilla/obj/dist/host/bin/mar -H firefox-mozilla-release -V 14.0.1 -i $f;
  /tools/python-2.6.5/bin/python \
    /builds/slave/rel-m-rel-updates/tools/release/signing/signtool.py \
    -t /builds/slave/rel-m-rel-updates/token -n /builds/slave/rel-m-rel-updates/nonce \
    -c /builds/slave/rel-m-rel-updates/tools/release/signing/host.cert \
    -H signing1.build.scl1.mozilla.com:9120 -H signing2.build.scl1.mozilla.com:9120 -f gpg -f mar \
    "$f"
done ) 2>&1 | tee logs/12.0-14.0.1/sign-partials.log

## Regenerate the snippets (1 min)
perl patcher2.pl --create-patchinfo \
  --app=firefox --brand=Firefox \
  --config=patcher-configs/mozRelease-branch-patcher2-758795.cfg \
  2>&1 | tee logs/12.0-14.0.1/generate_snippets.log

# Snippet verification (<1 min)
cd temp/firefox/12.0-14.0.1
bash /builds/slave/rel-m-rel-updates/tools/release/compare-channel-snippets.sh \
  aus2 release aus2.test  releasetest

# Tidy and upload mar files (3 min)
find ftp/firefox/nightly/14.0.1-candidates/build1 -type f -exec chmod 644 {} \; 
find ftp/firefox/nightly/14.0.1-candidates/build1 -type d -exec chmod 755 {} \;
rsync -nav -e 'ssh -oIdentityFile=~/.ssh/ffxbld_dsa' '--exclude=*complete.mar' \
  ftp/firefox/nightly/14.0.1-candidates/build1/update \
  ffxbld@stage.mozilla.org:/pub/mozilla.org/firefox/nightly/14.0.1-candidates/build1/
# then repeat without the -n

# Upload snippets (1 min)
rsync -av -e 'ssh -oIdentityFile=~/.ssh/auspush' --exclude=complete.txt \
 aus2.test/ \
 ffxbld@aus3-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-14.0.1-build1-test
rsync -av -e 'ssh -oIdentityFile=~/.ssh/auspush' --exclude=complete.txt \
 aus2/ \
 ffxbld@aus3-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-14.0.1-build1

# aus3-staging
backupsnip Firefox-14.0.1-build1-test
pushsnip   Firefox-14.0.1-build1-test

Forced update verify builders.

Generate release-cck-mozilla11/12 snippets

By duplicating the 13.0.1 release/releasetest snippets and modifying the bouncer product, see bug for more info.

# ffxbld@aus3-staging
cd /opt/aus2/snippets/staging
mkdir -p Firefox-14.0.1-build1-funnelcake11_12-test/Firefox
rsync -a --exclude=Linux* --exclude=betatest \
  Firefox-14.0.1-build1{,-funnelcake11_12}-test/Firefox/13.0.1/
for d in `find Firefox-14.0.1-build1-funnelcake11_12-test -mindepth 6 -type d`; do
  mv $d $d-cck-mozilla11
  rsync -a $d-cck-mozilla11/ $d-cck-mozilla12/
  sed -i -e 's/\-complete/-complete-funnelcake11/' \
            -e 's/-partial-13\.0\.1/-partial-13.0.1-funnelcake11/' $d-cck-mozilla11/*.txt
  sed -i -e 's/\-complete/-complete-funnelcake12/' \
            -e 's/-partial-13\.0\.1/-partial-13.0.1-funnelcake12/' $d-cck-mozilla12/*.txt
done
 
rsync -a Firefox-14.0.1-build1-funnelcake11_12{-test,}/
for d in `find Firefox-14.0.1-build1-funnelcake11_12 -mindepth 6 -type d`; do
  mv $d ${d/releasetest/release}
done

backupsnip Firefox-14.0.1-build1-funnelcake11_12-test
pushsnip   Firefox-14.0.1-build1-funnelcake11_12-test

Update verify
Reset reserved slaves
https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Tagging_through_Signing#Reset_reserved_slaves

Check permissions / AV scan
No problems, but will need to rerun after 12.0 - 14.0.1 partial generation

Push to internal mirrors
https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Push_to_releases_directory_.2F_internal_mirrors

There is extra work to do here for funnelcake, see the bug

Final verification
https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Final_Verification

Before Release Day
Run backupsnip for release snippets
https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Run_Backupsnip

Push index file to mirrors
https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Dealing_with_index.html_files

Push to external mirrors
https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Push_to_external_mirrors

Unblock firefox/releases/13.0.1-funnelcake/update as well firefox/releases/14.0.1/.

Release Day
Update XULRunner wiki page
https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Update_XULRunner_wiki_page

Check Throttling
https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Check_Throttling

Push to Release Channel (for beta releases and release releases)
https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Push_snippets

We'll have extra snippets to push for funnelcake, and the 12.0-14.0.1 partial. The correct order is 
 * release-cck-mozilla11, release-cck-mozilla12
 * main snippets
 * 12.0-14.0.1 partial

Update symlinks
https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Update_Symlinks

Remove index.html files
https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Dealing_with_index.html_files

Update browser choice website
https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Update_the_Browser_Choice_Website

Build 2 (Fennec only)
Preparing to start Automation
* Landed config update
* Retagged with FENNEC_14_0_1_{BUILD2,RELEASE}
* Updated, reconfiged.
* Ran release sanity, passed:
 cd /builds/buildbot/build1
 source bin/activae
 PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u bhearsum     -V 14.0.1 --branch mozilla-release --build-number 2 --release-config release-fennec-mozilla-release.py --products fennec      --dryrun localhost:9001
* Ran without dryrun to start:
 PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u bhearsum     -V 14.0.1 --branch mozilla-release --build-number 2 --release-config release-fennec-mozilla-release.py --products fennec  localhost:9001

Tag
No problems

Source
No problems

Build
Fennec
No problems

Fennec repacks
No problems

Android Signing
 # cltsign@signing1
 cd ~/signing-work/mozharness
 hg pull -u && hg up -C -r default
 python2.6 ~/signing-work/mozharness/scripts/sign_android.py --config-file signing/android_mozilla-release.py --platform android

Release Day
Publish Fennec to the Play Store (for beta releases and release releases)
Push the files (for beta releases and release releases)
https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Push_the_files

Play Store (for beta releases and release releases)
https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Play_Store

Build #	Branch, Tags	Changeset
1	MOBILE1401_2012071313_RELBRANCH, FENNEC_14_0_1_RELEASE, FENNEC_14_0_1_BUILD1	9c64e364f5df
1	GECKO1401_2012071313_RELBRANCH, FIREFOX_14_0_1_BUILD1 FIREFOX_14_0_1_RELEASE	e5728a4e106c
2	MOBILE1401_2012071316_RELBRANCH FENNEC_14_0_1_BUILD2 FENNEC_14_0_1_RELEASE	5ca6bd1f2bdf

Releases/Firefox 14.0.1/BuildNotes

Notes About Releasing

Bugs hit

Build Engineers

Signed-off Revision(s)

L10N changesets

Tags

Build data

Firefox

Fennec

Notes

Build 1

Diff mozconfigs

Set reserved slaves

Preparing to start Automation

E-mail Metrics

E-mail mirrors@mozilla.org

Edit rsync exclude files

Tag

Bouncer Submitter

Source

Build

Firefox

Firefox repacks

Fennec

Fennec repacks

XULRunner

Source

Build

Push to Mirrors

Partner Repack

Firefox

Android Signing

Checksums

Updates

Manually generate 13.0.1-14.0.1 partials

Adjust What's New in snippets

Generate 12.0 to 14.0.1 partial update for windows

Generate release-cck-mozilla11/12 snippets

Update verify

Reset reserved slaves

Check permissions / AV scan

Push to internal mirrors

Final verification

Before Release Day

Run backupsnip for release snippets

Push index file to mirrors

Push to external mirrors

Release Day

Update XULRunner wiki page

Check Throttling

Push to Release Channel (for beta releases and release releases)

Update symlinks

Remove index.html files

Update browser choice website

Build 2 (Fennec only)

Preparing to start Automation

Tag

Source

Build

Fennec

Fennec repacks

Android Signing

Release Day

Publish Fennec to the Play Store (for beta releases and release releases)

Push the files (for beta releases and release releases)

Play Store (for beta releases and release releases)

Navigation menu

Search