Build Engineers

Signed-off Revision(s)

Build data

Build #	Type	Build ID	Build machine	Time to build
1	Linux	20101122045408	moz2-linux-slave14	1 hrs, 22 mins, 30 secs
	Mac	20101122044323	bm-xserve17	1 hrs, 37 mins, 28 secs
	Windows	20101122053531	mw32-ix-slave06	1 hrs, 41 mins, 46 secs
2	Linux	20101130065329	moz2-linux-slave15	1 hrs, 17 mins, 27 secs
	Mac	20101130064313	bm-xserve17	1 hrs, 35 mins, 58 secs
	Windows	20101130064308	mw32-ix-slave06	1 hrs, 46 mins, 5 secs
3	Linux	20101203074139	mv-moz2-linux-ix-slave03	37 mins, 16 secs
	Mac	20101203074432	bm-xserve17	1 hrs, 38 mins, 20 secs
	Windows	20101203075014	mw32-ix-slave06	1 hrs, 57 mins, 32 secs

Build 1

Done on pm03. Setup notes:

No clobberer necessary because this is the first 0.8.0 release on this branch.
landed config updates, including l10n-changesets
tagged buildbot-configs, buildbotcustom, compare-locales, and tools with FIREFOX_3_6_12_BUILD1 and FIREFOX_3_6_12_RELEASE
updated /tools/buildbotcustom/buildbotcustom, /builds/buildbot/configs, and /builds/buildbot/builder_master/tools to latest
reconfiged master
Left tree open, 0.8.0 automation doesn't require that it closes
kicked off automation with:

buildbot sendchange --username=bhearsum --master=localhost:9010 --branch=releases/mozilla-1.9.2 --revision=FIREFOX_3_6_13_RELEASE -p release_config:mozilla/release-firefox-mozilla-1.9.2.py gogogo

Tag

'pl' locale failed when 'hg out' got a bad response from the HG server. (FIXME we should be retrying when any hg operation fails).

Fixed manually by running the following on mv-moz2-linux-ix-slave11, as cltbld:

cd /builds/slave/release-mozilla-1.9.2-tag/build/pl
hg push --new-branch -e 'ssh -l ffxbld -i ~/.ssh/ffxbld_dsa' ssh://hg.mozilla.org/releases/l10n-mozilla-1.9.2/pl

Manually started source, xulrunner_source, _build and _xulrunner_build for all platforms, filling in the form as follows:

Branch: releases/mozilla-1.9.2
Revision: FIREFOX_3_6_13_RELEASE
Property 1 Name: release_config
Property 1 Value: mozilla/release-firefox-mozilla-1.9.2.py

Source / XULRunner Source

No Problems

Build

No problems. Removed en-US.xpi from stage

L10n Repack

Triggered manually, see Tag section for form details.

Failed a few times because the scripts were trying to use staging-only variables. Landed a few bustage fixes before it got working again. Patches are in bug 613944.

Lots of individual repacks failed when uploading. Found them by searching for "ssh_exchange_identification: Connection closed by remote host" in the logs. To fix, did the following for each failed locale (on the machine that tried to build it):

export UPLOAD_USER=ffxbld
export MOZ_OBJDIR=obj-l10n
export UPLOAD_SSH_KEY="~/.ssh/ffxbld_dsa"
export UPLOAD_HOST=stage.mozilla.org
export POST_UPLOAD_CMD="post_upload.py -p firefox -n 1 -v 3.6.13 --release-to-candidates-dir"
export UPLOAD_TO_TEMP=1
export MOZ_PKG_PRETTYNAMES=1
cd /builds/moz2_slave/release-mozilla-1.9.2-$platform_repack_$n/build/mozilla-1.9.2/obj-l10n/browser/locales
export AB_CD=$locale
make l10n-upload-$locale

Did the following on stage to make sure we had all of the locales:

cd /home/ftp/pub/firefox/nightly/3.6.13-candidates/build1
ls unsigned/win32 | grep -v xpi | wc -l
ls mac | grep -v xpi | wc -l
ls linux-i686 | grep -v xpi | wc -l

Output for each should be 76, which it was.

Partner Repack

No problems. Signing happened at the same time as other builds.

XULRunner Build

No problems

Unittests

Mac: All green
Linux: All green
Win32: All green except for mochitests, which ran all test successfully and then hung at shutdown. Retriggered it, went green.

Signing

Removed en-US.xpi from candidates directory. No problems, followed CombinedSigning doc.

Got "Signature verification succeeded" mail.

L10n Verify

Many string changes across many platforms/locales. All seemed legitimate, didn't dig further.

Updates

3.6* -> 3.6.13

Generation

First attempt failed when the Linux 'id' MAR failed to download. Dug in a bit more and found that all locales I had recovered manually were missing MARs because I forgot to set MOZ_MAKE_COMPLETE_MAR=1 when uploading. To fix
re-did the l10n uploads w/ MOZ_MAKE_COMPLETE_MAR
resigned everything.
clobbered updates builddir
restarted updates from the beginning
No other issues

Update Verify

Linux: PASS
Windows: PASS
Mac: Complained that there was no partial update for 3.6.4's 'ca' locale. Manually checking of the link did not reproduce, likely a fluke. Re-run update verify for confirmation.
- No issues on the re-run

Socorro product data updates

Added '3.6.13' and '3.6.14pre' to crash-stats with this admin page.

Update Bouncer

Used "Force Build" to start the "release-mozilla-1.9.2-bouncer_submitter" builder.

First attempt failed with the error: "Cannot retrieve credentials file: BuildSlaves.py"
- Noticed that tuxedoUsername/tuxedoPassword were missing from BuildSlaves.py on pm01/pm03.
After adding the entries I used "Force Build" again. No problems this time.

Stage EU Ballot Builds

Using the EU Ballot builds generated by the build machines
No signing necessary as these were signed earlier
Kev looked over them before staging

Once Kev verified them, ran the following to get them ready for the mirror push:

# ffxbld@stage
cd /home/ftp/pub/firefox/nightly/3.6.13-candidates/build1
mv -v partner-repacks/euballot/win32 win32-EUballot

Ideally, the partner repack scripts or signing would upload this to this location from the git go. Filed bug 614227 to track this.

Push to Beta Channel

As cltbld@aus2-staging:

~/bin/backupsnip 20101122-Firefox-3.6.13-beta
~/bin/pushsnip 20101122-Firefox-3.6.13-beta

Build 2

Done on pm03. Setup notes:

Set a clobber for pm03, 'Any release', 'Any builder' with clobberer.
- Noticed that after clobbering, all of the release builders showed up in the table. Filed bug 615192 on the issue.
Tagged buildbot-configs, buildbotcustom, and tools with FIREFOX_3_6_13_RELEASE and FIREFOX_3_6_13_BUILD2. compare-locales will be tagged by the tagging step this time.
Updated /tools/buildbotcustom/buildbotcustom, /builds/buildbot/buildbot-configs, and /builds/buildbot/builder_master/tools to latest
reconfiged master
Ran release_sanity.py with --dryrun:

PYTHONPATH=.:tools/lib/python python tools/buildbot-helpers/release_sanity.py -u bhearsum -V 3.6.13 --branch mozilla-1.9.2 --build-number 2 --dryrun localhost:9010

After it passed, ran it for real:

PYTHONPATH=.:tools/lib/python python tools/buildbot-helpers/release_sanity.py -u bhearsum -V 3.6.13 --branch mozilla-1.9.2 --build-number 2 localhost:9010

Tag

Cancelled the first run because it merged other changesets, and had revision set wrong.
- Catlee said that one or both of bug 613429 and bug 614946 could be to blame.
Re-ran release_sanity.py, revision was set correctly this time. No further issues.

Source/XULRunner Source

No problems

Build

No problems

L10n Repack

No problems

Partner Repack

No problems

XULRunner Build

No problems.

Sign

Removed en-US.xpi from candidates directory.

Followed instructions on CombinedSigning page.

No problems.

L10n Verify

Many string changes across many platforms/locales. All seemed legitimate, didn't dig further.

Update Generation

First attempt failed with:

Traceback (most recent call last):
  File "/builds/slave/release-mozilla-1.9.2-updates/tools/release/generate-candidate-build-updates.py", line 199, in <module>
    sys.exit(main())
  File "/builds/slave/release-mozilla-1.9.2-updates/tools/release/generate-candidate-build-updates.py", line 194, in main
    options.verbose)
  File "/builds/slave/release-mozilla-1.9.2-updates/tools/release/generate-candidate-build-updates.py", line 55, in createSnippets
    verbose)
TypeError: getShippedLocales() takes at most 6 arguments (7 given)

Tracked in bug 615668. After landing the patch, backed out update verify changes, retagged build/tools with FIREFOX_3_6_13_RELEASE and FIREFOX_3_6_13_BUILD2, clobbered the updates builder, and used "Force Build" to re-start updates.

No further problems.

Update Verify

All platforms: PASS

Stage EU Ballot Builds

Using the EU Ballot builds generated by the build machines
No signing necessary as these were signed earlier
Kev looked over them before staging

Once Kev verified them, ran the following to get them ready for the mirror push:

# ffxbld@stage
cd /home/ftp/pub/firefox/nightly/3.6.13-candidates/build2
mv -v partner-repacks/euballot/win32 win32-EUballot

Ideally, the partner repack scripts or signing would upload this to this location from the git go. Filed bug 614227 to track this.

Push to beta channel

~/bin/backupsnip 20101130-Firefox-3.6.13-beta
~/bin/pushsnip 20101130-Firefox-3.6.13-beta

ashughes reported that he was not receiving updates for 3.6.13build1 on Mac. After digging into it I discovered that this was because the 3.6.13 Mac builds use "Darwin_x86-gcc3-u-ppc-i386" instead of "Darwin_Universal-gcc3" in the update URL. To workaround the issue for now, I created a symlink in the live snippet directory:

# cltbld@aus2-staging
cd /opt/aus2/incoming/3/Firefox/3.6.13
ln -s Darwin_x86-gcc3-u-ppc-i386 Darwin_Universal-gcc3

The long term fix is getting patcher tools to support the new OS names, tracked in bug 583671, which needs to be fixed before 3.6.14.

At LegNeato's request, we completely unthrottled the 3.6 beta channel. Tracked in bug 615902.

Build 3

Done on pm03. Setup notes:

Set a clobber for pm03, 'Any release', 'Any builder' with clobberer.
Tagged buildbot-configs, buildbotcustom, and tools with FIREFOX_3_6_13_RELEASE and FIREFOX_3_6_13_BUILD3.
Updated /tools/buildbotcustom/buildbotcustom, /builds/buildbot/buildbot-configs, and /builds/buildbot/builder_master/tools to latest
reconfiged master
Ran release_sanity.py with --dryrun:

PYTHONPATH=.:tools/lib/python python tools/buildbot-helpers/release_sanity.py -u bhearsum -V 3.6.13 --branch mozilla-1.9.2 --build-number 3 --dryrun localhost:9010

After it passed, ran it for real:

PYTHONPATH=.:tools/lib/python python tools/buildbot-helpers/release_sanity.py -u bhearsum -V 3.6.13 --branch mozilla-1.9.2 --build-number 3 localhost:9010

Tag

First attempt failed because of bug 614946. Re-run the second release_sanity.py command to get things going. No other problems.

Source / XULRunner Source

No problems

Builds

No problems

L10n Repacks

Partner Repacks

Linux partner repacks failed to upload with the following error:

rsync: mkdir "/pub/mozilla.org/firefox/nightly/3.6.13-candidates/build3/unsigned/partner-repacks" failed: No such file or directory (2)
rsync error: error in file IO (code 11) at main.c(577) [Receiver=3.0.7]
rsync: connection unexpectedly closed (8 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at /SourceCache/rsync/rsync-30/rsync/io.c(359)

To fix, did the following manually on bm-xserve09:

cd /builds/slave/release-mozilla-1.9.2-linux_partner_repack/partner-repacks/scripts/repacked_builds/3.6.13/build3
rsync -av -e "ssh -oIdentityFile=~/.ssh/ffxbld_dsa" . ffxbld@stage-old.mozilla.org:/pub/mozilla.org/firefox/nightly/3.6.13-candidates/build3/unsigned/partner-repacks

Filed bug 616512 on the issue.

XULRunner Builds

No problems

Signing

Removed en-US.xpi from candidates directory

To speed up signing, ran the following prior to all repacks being completed:

PRODUCT=firefox
VERSION=3.6.13
BUILD=3
TAG=FIREFOX_3_6_13_RELEASE
REPO='releases/mozilla-1.9.2'
EMAIL=bhearsum@mozilla.com
KEYDIR=d:/2010-keys
cd ~/hg-tools
hg pull && hg up -r ${TAG}
mkdir ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
make df verify-gpg-passphrase setup verify-signcode-passphrase PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
# download target run multiple times to get things as they complete
make download PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1

Once all the l10n and partner repacks finished, ran the following in the same window to do the signing:

# Need to download one more time to make sure we've got everything
make download PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
make stubs sign2 stage verify fake-upload PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1

And then to upload:

make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL}

Got "Signature Verification Succeeded" mail later.

L10n Verify

Updates

First attempt failed because the updates directory hadn't been clobbered. Filed bug 616581 on the issue. To fix, did the following:

Clobbered /builds/slave/release-mozilla-1.9.2-updates on mv-moz2-linux-ix-slave11 by hand
Backed out update verify changes
Used "Force Build" to restart the updates builder.

Update Verify

Linux: PASS
Windows: PASS
Mac: Complained that there was no partial update for 3.6b2's 'tr' locale. Manually checking of the link did not reproduce, likely a fluke. Re-run update verify for confirmation.

Push to Beta Channel

~/bin/backupsnip 20101203-Firefox-3.6.13-beta
~/bin/pushsnip 20101203-Firefox-3.6.13-beta

Build #	Tag	Changeset
1	FIREFOX_3_6_13_BUILD1 FIREFOX_3_6_13_RELEASE	af7a49ca89b2

Releases/Firefox 3.6.13/BuildNotes

Build Engineers

Signed-off Revision(s)

Tags

Build data

Build 1

Tag

Source / XULRunner Source

Build

L10n Repack

Partner Repack

XULRunner Build

Unittests

Signing

L10n Verify

Updates

3.6* -> 3.6.13

Generation

Update Verify

Socorro product data updates

Update Bouncer

Stage EU Ballot Builds

Push to Beta Channel

Build 2

Tag

Source/XULRunner Source

Build

L10n Repack

Partner Repack

XULRunner Build

Sign

L10n Verify

Update Generation

Update Verify

Stage EU Ballot Builds

Push to beta channel

Build 3

Tag

Source / XULRunner Source

Builds

L10n Repacks

Partner Repacks

XULRunner Builds

Signing

L10n Verify

Updates

Update Verify

Push to Beta Channel

Navigation menu

Search