Releases/Firefox 3.0.3:BuildNotes

Build Engineers

joduinn
Tracking release bug

Bonsai queries

Updated CVS Tags devmo page.

Tags

Build 1:

Module	Branch	Tag	Pull date
cvsroot/mozilla	HEAD	FIREFOX_3_0_3_BUILD1	2008-08-27 01:31 PDT
l10n/l10n	HEAD	FIREFOX_3_0_3_BUILD1	2008-08-27 01:31 PDT

Build data

Type	Build ID	SHA1	Push date	Build machine
[Windows installer/zip]				fx-win32-1.9-slave2
[Mac compressed]				fx-mac-1.9-slave2
[Linux compressed]				fx-linux-1.9-slave2

Notes

Build 1

Bootstrap Tag: RELEASE_AUTOMATION_M11

Setup before starting:

• Updated master.cfg, bootstrap.cfg. Retagged bootstrap.cfg.
• On the slaves:
  • ran 'DISPLAY=:0 xhost +' to make sure linux AliveTest works.
  • remove /builds/verify/firefox-3.0.1
  • update tinderbox to RELEASE_AUTOMATION_M11
    • For win32:

cd /e/builds/tinderbox/mozilla/tools/tinderbox
cvs up -r RELEASE_AUTOMATION_M11
for dir in fx19rel fx19l10nrel xr19rel; do 
  cd /e/$dir; 
  for f in `ls *.pl *.html INSTALL README | grep -v tinder-config.pl`; do 
    cp -pv /e/builds/tinderbox/mozilla/tools/tinderbox/$f .;
  done
  cp -pv post-mozilla-rel.pl post-mozilla.pl
done

• • On the linux slave, removed:
    • /builds/updates/firefox-3.0.1 and /builds/updates/3.0.2-test
    • /builds/source/*
    • /data/cltbld/thunderbird-3.0a2*
  • On the mac slave, removed:
    • /builds/tinderbox/Tb-Mozilla1.9-Release/Darwin_8.8.4_Depend
    • /builds/tinderbox/Tb-Mozilla1.9-l10n-Release/Darwin_8.8.4_Depend
    • /builds/tinderbox/Xr-Mozilla1.9-Release/Darwin_8.8.4_Depend

• Space on slaves before starting:
  • fx-linux-1.9-slave2: 11G on /builds
  • fx-mac-1.9-slave2: 24G on /
  • fx-win32-1.9-slave2: 11G on d:, 2G on e: (disk heavy work is done on d, eg update_verify. build/repack is done on e, but mostly just overwrites existing data)

• Gulped, and kicked off automation:

buildbot sendchange --username=joduinn --master=localhost:9989 -m"Firefox 3.0.3build1 release" go

Tag

• Automated, no problems.

Source

• Automated, no problems.

Build & Repack

• mac: Automated, no problems.
• linux: build went fine, but then "AliveTest" failed because I missed the DISPLAY setting instruction above. This has not been done since the colo power outage a couple of days ago. Fixed missing DISPLAY, restarted build and this time linux build passed.
• win32: buildbot master crashed out during l10n repack. Manually restarted, patched master config to restart at win32 repack and continue onwards. This worked fine.

Sign

instructions

Signing failed out on 'de' with:

 Error: Invalid timestamp http address
 Error: TimeStampign Failed. Result = 80070001, (-2147024895)

To work-around, did the following (from https://bugzilla.mozilla.org/show_bug.cgi?id=415966#c25):

mkdir signed-build6-partial unsigned-build6-partial
cp signed-build1/*en-US* signed-build1-partial/
cp unsigned-build1/*en-US* unsigned-build1-partial/
# copied everything from 'de' -> 'zh-TW' from unsigned-build1 to unsigned-build1-partial
perl ./signing/sign-release.pl -o signed-build${BUILD}-partial -M dir -d unsigned-build${BUILD}-partial -a ${PRODUCT} --release ${TAG} 2>&1 | tee -a win32_signing_build${BUILD}.log
# to merge the signed builds back together
mv signed-build6-partial/ signed-build6/

...and again when failed on 'fy'.

# then push the builds back to stage (instructions on the signing doc)

L10nVerify

• Automated, no problems.

Generate Updates

Need to bring people who already updated to 3.0.2 build5 up to build6, so do this (after the automated run)

# connect as cltbld@fx-linux-1.9-slave2
cd /builds/updates/firefox-3.0.2/patcher/temp/firefox/3.0.1-3.0.2
for dir in aus2.test aus2.beta aus2; do
  # copy complete snippets from 3.0.1 into a new dir for 3.0.2
  mkdir -p b5-b6-${dir}/Firefox
  rsync -a --exclude 'partial.txt' ${dir}/Firefox/3.0.1/ b5-b6-${dir}/Firefox/3.0.2/
  # use the buildid's from build5
  mv b5-b6-${dir}/Firefox/3.0.2/Darwin_Universal-gcc3/2008070206  b5-b6-${dir}/Firefox/3.0.2/Darwin_Universal-gcc3/2008090512
  mv b5-b6-${dir}/Firefox/3.0.2/Linux_x86-gcc3/2008070206         b5-b6-${dir}/Firefox/3.0.2/Linux_x86-gcc3/2008090512
  mv b5-b6-${dir}/Firefox/3.0.2/WINNT_x86-msvc/2008070208         b5-b6-${dir}/Firefox/3.0.2/WINNT_x86-msvc/2008090514
done
rsync -a -e 'ssh -i ~/.ssh/aus' b5-b6-aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080919-Firefox-3.0.2build5-build6-test
rsync -a -e 'ssh -i ~/.ssh/aus' b5-b6-aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080919-Firefox-3.0.2build5-build6-beta
rsync -a -e 'ssh -i ~/.ssh/aus' b5-b6-aus2/      cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080919-Firefox-3.0.2build5-build6

Then go to aus2-staging and push 20080919-Firefox-3.0.2build5-build6-test live.

Publish Updates to Test Channels (betatest & releasetest)

• Automated, no problems.

Update Verify

• linux no problems
• win32 expected problems (known checksum dif bug)
• mac expected problems (added/dropped locales)

Stage

• Automated, no problems.

Push to beta channel

Backup on aus2-staging doing:

 # put snippets on beta
 $ sudo su - cltbld
 # make sure using latest version of scripts in mozilla/tools/release/bin/
 $ cd bin
 $ cvs update . 
 $ cd /opt/aus2/snippets/staging
 # note the required parameter must match what will be used with pushsnip below. 
 $  time ~/bin/backupsnip 20080925-Firefox-3.0.3-beta
 Running /bin/tar cfvj /opt/aus2/snippets/backup/20080925-3-pre-20080925-Firefox-3.0.3-beta.tar.bz2 .
 real    37m9.378s
 user    0m45.106s
 sys     1m41.298s
 $
 $ time ~/bin/pushsnip 20080925-Firefox-3.0.3-beta
real    2m15.445s
user    0m0.341s
sys     0m19.679s

Sign Installers

Done manually using these installer-signing-instructions here

• complete stage-merged:

# on stage
cd /data/cltbld/firefox-3.0.3/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/

• Create MD5 and SHA1 checksum files

# on stage
cd /data/cltbld/firefox-3.0.3/stage-merged/
~/bin/checksum-files .

• Fix permissions & ownership (on the two SUM files, and the detached sigs)

chown -R cltbld:firefox .
chmod 644 *SUMS

Update Bouncer

• Manually added entries to download.mozilla.org

Push to mirrors

• push the stage-merged directory to the releases area:

# on stage
rsync -av /data/cltbld/firefox-3.0.2/stage-merged/ /home/ftp/pub/firefox/releases/3.0.2/

Final Verification

• Verify that releasetest points to valid bouncer links:

# this can be run from anywhere
cvs co mozilla/testing/release
cd mozilla/testing/release/updates
cat moz19-firefox-*.cfg | grep -v major | sed 's/betatest/releasetest/' | grep -v 2.0a | grep -v 2.0b > update.cfg
./verify.sh -t update.cfg 2>&1 | tee quickVerify.log

• Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):

grep HTTP quickVerify.log | grep -v 200 | grep -v 302

• • 23 occurrences of 404s

• Before pushing final updates,verify that "release" and "releasetest" channel match:

# on aus2-staging
$ cd 20080917-Firefox-3.0.2
$ find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20080917-Firefox-3.0.2-test/$a");'
$

Publish Updates to Release Channel

While waiting for formal "go", do:

$ time ~/bin/backupsnip 20080917-Firefox-3.0.2
real    33m24.621s
user    0m38.584s
sys     0m41.689s

$ time ~/bin/pushsnip 20080917-Firefox-3.0.2
real    0m17.352s
user    0m0.027s
sys     0m1.464s
$ time ~/bin/pushsnip 20080919-Firefox-3.0.2build5-build6
real    0m8.355s
user    0m0.004s
sys     0m0.329s

NB: Be sure to push both 20080917-Firefox-3.0.2 and 20080919-Firefox-3.0.2build5-build6 directories.

Release

• On stage.m.o, do the following:
  • edit the exclude file /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude to add the new release.

• • Update "latest" symlinks

$ cd /pub/mozilla.org/firefox/releases/
$ mv latest-3.0 latest-3.0.old
$ ln -s 3.0.2 latest-3.0
$ rm latest-3.0.old