Latest revision as of 00:05, 14 March 2011

Notes About Releasing

Please update the Release:Primer for future releases (bug fixes, changes to automation) as needed

Build Engineers

Signed-off Revision(s)

Build 1: 290712e55ade

L10N changesets

Retrieved the l10n-changesets as follows:

wget -O l10n-changesets_mozilla-2.0 https://l10n-stage-sj.mozilla.org/shipping/l10n-changesets?ms=fx4.0rc1

Hit "ship it" on Firefox 4.0rc1 page (https://l10n-stage-sj.mozilla.org/shipping/milestones)

Build data

Build #	Type	Build ID	Build machine	Time to build
1	Linux [compressed .tar.bz2]	20110303140816	linux-ix-slave19	57min2s
	Linux64 [compressed .tar.bz2]	20110303144013	moz2-linux64-slave05	1h36min23s
	Mac [.dmg]	20110303140758	moz2-darwin10-slave13	4h27min42s
	Win32 [.exe/.zip]	20110303194838	w32-ix-slave30	hours and hours because of manual kicking

Build 1

We're on pm01 for this release.

Setup

Set a clobber

Set a clobber for pm01, any release, any builder with clobberer.

Sync repositories

We're building RCs out of the mozilla-2.0 and l10n-mozilla-2.0 repositories, and we're responsible for syncing the latest from mozilla-central/l10n-central to them. To do this, did the following on an idle slave, after opening the mozilla-2.0 tree:

cd /builds/slave
mkdir rc1-sync
cd rc1-sync
wget -O- http://hg.mozilla.org/mozilla-central/raw-file/tip/browser/locales/shipped-locales  | grep -v en-US | cut -f1 -d" " > locales.txt
hg clone http://hg.mozilla.org/mozilla-central
cd mozilla-central
hg push -e 'ssh -i ~/.ssh/ffxbld_dsa' ssh://ffxbld@hg.mozilla.org/releases/mozilla-2.0
for i in `cat locales.txt`; do hg clone http://hg.mozilla.org/l10n-central/$i; done
for i in `cat locales.txt`; do hg -R $i push -e 'ssh -i ~/.ssh/ffxbld_dsa' ssh://ffxbld@hg.mozilla.org/releases/l10n-mozilla-2.0/$i; done

Only a few locales had changesets pushed, because the l10n-mozilla-2.0 were cloned earlier in the day.

Reserve Slaves

On pm01, did the following, to reserve enough fast slaves for the release:

cd /builds/buildbot/builder_master1
echo 6 > reserved_slaves_pm01

Move IX machines

Because of 638309 we had to connect some SCL-based IX machines to pm01 for the duration of this release. Moved the following:

linux-ix-slave18 - 23
w32-ix-slave27 - 32

Land Configs

Landed release config updates
Also added "force_plist_reload" to mozilla-2.0's patcher config, to ensure no strangeness with that file like we had in b9 througb 12.

Tag Repositories

Tagged build/tools, buildbotcustom, and buildbot-configs with FIREFOX_4_0rc1_RELEASE and FIREFOX_4_0rc1_BUILD1

Update Master

Updated /tools/buildbotcustom/buildbotcustom, /builds/buildbot/builder_master/tools and /builds/buildbot/configs on pm01 to the latest of the production branches. Afterwards, reconfiged with the following command:

cd /builds/buildbot/builder_master1
make reconfig && tail -F twistd.log | grep configuration

Run release_sanity.py

PYTHONPATH=/builds/buildbot/builder_master1:/builds/buildbot/builder_master1/tools/lib/python python /builds/buildbot/builder_master1/tools/buildbot-helpers/release_sanity.py -u bhearsum -V 4.0rc1 --branch mozilla-2.0 --build-number 1 -c release-firefox-mozilla-2.0.py --dryrun localhost:9010

Kick it off

PYTHONPATH=/builds/buildbot/builder_master1:/builds/buildbot/builder_master1/tools/lib/python python /builds/buildbot/builder_master1/tools/buildbot-helpers/release_sanity.py -u bhearsum -V 4.0rc1 --branch mozilla-2.0 --build-number 1 -c release-firefox-mozilla-2.0.py --bypass localhost:9010

Tag

No problems.

Bouncer Update

Triggered automatically after tagging, no issues.

Start autosign

Once mozilla-2.0 finished tagging, used these instructions to start autosign

Autosign failed "Incorrect number of .asc files" due to extra index.html file on stage. Added '-not -name "*.html"' to TOTAL_FILE macro and reran "make verify fake-upload postsing ..."

Source

No problems

Build/Repack

Linux: no problems with en-US
- L10n repacks failed the first time because I forgot to update the mozconfigs for the l10n-mozilla-2.0 repositories. They failed a second time because I forgot to retag buildbot-configs after landing the fix.
- Because of failed repacks, "linux_repack_complete" builder didn't fire. Used "Force Build" to trigger it manually.

Linux 64-bit: no problems with en-US or repacks

Win32
- lost connection to w32-ix-slave29 during linking, buildbot tried to retry on w32-ix-slave30 automatically, but couldn't connect to it. catlee rebooted slave30, and then updated the db so the request would be retried again. slave29 picked it up a second time.
- 2nd build on slave29 failed, automatically retried and picked up by slave30 this time. catlee moved all w32-ix slaves to buildbot-master1:9010
- hit rebuild on build#1 so we could get a build going on a VM at the same time in case the one on the IX machine dies again
- final build attempt on IX through buildbot died because bhearsum tried to trick the linker into being verbose, but caused a bad invocation instead. Kicked it through manually by doing the following:

# In MSYS shell through VNC on w32-ix-slave30
cd /e/builds/moz2_slave/rel-2.0-w32-bld/build
export MOZ_OBJDIR=obj-firefox
export MOZ_PKG_PRETTYNAMES=1
export MOZ_PKG_VERSION=4.0rc1
make -f client.mk profiledbuild MOZ_BUILD_DATE=20110303194838
cd obj-firefox
make buildsymbols
export POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py
export SYMBOL_SERVER_HOST=dm-symbolpush01.mozilla.org
export SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_ffx/
export SYMBOL_SERVER_SSH_KEY="/c/Documents and Settings/cltbld/.ssh/ffxbld_dsa"
export SYMBOL_SERVER_USER=ffxbld
make uploadsymbols
make package
make installer
make -C obj-firefox/tools/update-packaging
cd ../
python build/config/printconfigsetting.py build/obj-firefox/dist/bin/application.ini App BuildID > build/obj-firefox/dist/win32_info.txt
export POST_UPLOAD_CMD="post_upload.py -p firefox -n 1 -v 4.0rc1 --release-to-candidates-dir"
export UPLOAD_EXTRA_FILES=win32_info.txt
export UPLOAD_HOST=stage.mozilla.org
export UPLOAD_SSH_KEY=~/.ssh/ffxbld_dsa
export UPLOAD_TO_TEMP=1
export UPLOAD_USER=ffxbld
cd build/obj-firefox
make upload

Manually sent "unsigned win32 en-US build available" mail
Manually forced the 6 win32 l10n builders with the following properties set:
- release_config: mozilla/release-firefox-mozilla-2.0.py
- script_repo_revision: FIREFOX_4_0rc1_RELEASE

OSX: no problems with en-US or repacks

Added index.html the firefox/nightly/4.0rc1-candidates/build1/ to redirect the curious/obsessive ftp polling types.

Returned linux-ix-slave18 - 23 to bm04, since they had gotten hung anyway.

Unittests & Talos

None of these ran - bug 641399.

Signing

No problems, autosign worked fine. Got "Signature verification succeeded" mail.

Unreserve Slaves

On pm01, did the following:

cd /builds/buildbot/builder_master1
echo 0 > reserved_slaves_pm01

L10N Verification

post_sign bits weren't triggered. FtpPoller was polling http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/4.0rc1-candidates/build1/ which instead of showing a directory listing was redirecting to http://www.mozilla.com/en-US/firefox/comingsoon/

Forced.

Results:

All Platforms:
- "fa" only in RC1.
  - This is expected because it's a new locale as of this release.
- Big spew for 'br' locale, which had a big dictionary upgrade in bug 637423
- Small updates for various other locales

Nothing else of note, PASS.

Updates

post_sign bits weren't triggered. FtpPoller was polling http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/4.0rc1-candidates/build1/ which instead of showing a directory listing was redirecting to http://www.mozilla.com/en-US/firefox/comingsoon/

Forced.

Failed a few times because it couldn't free enough space
Failed another time because it couldn't read the a buildid properly.
- Problem turned out to be that the win32_info.txt wasn't formatted correctly, which was fallout from manually forcing it through earlier. After fixing it to be in the correct format (buildid=2011xxxxxxxxxx), forced another run.
- This run succeeded

Update Verify

Manually forced update verify builders. Results:

All platforms/locales:
- Update channel differences ("beta" vs "release")
- Test Pilot removal listed in removed-files

The update channel change, and test pilot listing in removed-files are both expected and innocuous. Nothing else of note, so this is a PASS.

Permissions Check

Triggered through force build with the following properties:

release_config: mozilla/release-firefox-mozilla-2.0.py
script_repo_revision: FIREFOX_4_0rc1_RELEASE

Virus Scan

Triggered through force build with the following properties:

release_config: mozilla/release-firefox-mozilla-2.0.py
script_repo_revision: FIREFOX_4_0rc1_RELEASE

No issues found.

XULRunner Builds

Had to land a follow-up patch to enable XULRunner builds. Triggered them through "Force Build"

Linux 32-bit: No problems
Linux 64-bit: No problems
Windows: No problems
Mac: Failed to compile. Filed bug 638921 on the issue.
- A rebuild succeeded compilation, but failed due to bug 634809
  - Worked around this manually with:

# cltbld@moz2-darwin10-slave17
cd /builds/slave/rel-2.0-xr-osx64-bld/build/obj-firefox/x86_64/dist/sdk
scp -i ~/.ssh/xrbld_dsa \
 xulrunner-2.0.en-US.mac-x86_64.sdk.tar.bz2 \
 xrbld@stage.mozilla.org:/pub/mozilla.org/xulrunner/nightly/2.0-candidates/build1/sdk/

All files were uploaded to "2.0-candidates" instead of "2.0rc1-candidates" because of bug 639888. Manually did the following to fix:

# xrbld@stage
cd /home/ftp/pub/xulrunner/nightly
mv 2.0-candidates 2.0rc1-candidates

XULRunner Signing

#!/bin/bash
set -x
set -e

PRODUCT=xulrunner
VERSION=2.0rc1
BUILD=1
TAG=FIREFOX_4_0rc1_RELEASE
REPO="releases/mozilla-2.0"
EMAIL=@mozilla.com
KEYDIR=d:/2010-keys

cd ~/hg-tools
hg pull
hg up -r ${TAG}

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .

make setup PRODUCT=${PRODUCT} VERSION=${VERSION} \
  BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  --exclude=*.txt --exclude=*-symbols.zip \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/ \
 unsigned-build${BUILD}
rsync -av --exclude=unsigned unsigned-build${BUILD}/ \
 signed-build${BUILD}/
rsync -av unsigned-build${BUILD}/unsigned/ \
 signed-build${BUILD}/

make checksum-files create-sigs stage verify-sigs \
  PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} \
  REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1

rmdir signed-build${BUILD}/contrib{,-localized}

rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  signed-build${BUILD}/ \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/

Socorro product data updates

Someone else had already added "4.0". I added "4.0pre".

Push to Mirrors

Forced the "release-mozilla-2.0-push_to_mirrors" builder with the following properties:

release_config: mozilla/release-firefox-mozilla-2.0.py
script_repo_revision: FIREFOX_4_0rc1_RELEASE

Notify AV vendors

Notify vendors using instructions in bug 568884#c15
this done out of sequence as missing from build notes, and manually caught. Not sure if AV vendors will have enough time to scan *today* as push-to-mirrors already well in progress. Need to do this earlier in process, see FF3.6.14, FF3.6.15 for examples.

Notify Mirrors

Normally this is done well in advance of pushing, but I forgot to do it until well after. Sent short mail to mirrors@mozilla.org informing them that 4.0rc1 was pushed.

Push Beta Channel Snippets

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20110303-Firefox-4.0rc1
~/bin/pushsnip 20110303-Firefox-4.0rc1

Send mail to metrics

Firefox,firefox,firefox,4.0,4.0
Firefox,firefox,firefox,4.0pre,4.0

Release

# cltbld@stage.mozilla.org
cd /pub/mozilla.org/zz/

Updated rsync exclude files from the above dir to exclude b12

Push XULRunner to mirrors

# xrbld@stage
rsync -av --exclude=*.txt --exclude=*unsigned*  --exclude=*-symbols.zip \
  /pub/mozilla.org/xulrunner/nightly/2.0rc1-candidates/build1/ \
  /pub/mozilla.org/xulrunner/releases/2.0rc1/

Edited cltbld@stage:/pub/mozilla.org/zz/rsyncd-mozilla-{pre,}releases.exclude and added:

- xulrunner/releases/2.0b12

Update XULRunner entries on wiki

Updated the links in these two wiki pages for the version change:

Remove index.html

After release, remove the index.html:

# ffxbld@stage
cd /pub/mozilla.org/firefox/releases/4.0rc1
rm index.html

Build #	Tag	Changeset
1	GECKO20_2011030313_RELBRANCH	290712e55ade
1	FIREFOX_4_0rc1_BUILD1 FIREFOX_4_0rc1_RELEASE	5f8f494a4c29

Releases/Firefox 4.0rc1/BuildNotes: Difference between revisions