Firefox 3.0b4:BuildNotes: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(91 intermediate revisions by 4 users not shown)
Line 1: Line 1:
Scratch pad of unusual TODO's for this release:
* [http://developer.mozilla.org/en/docs/Building_with_Profile-Guided_Optimization Profile Guided Optimisation]
** we need at least rev1.383 of tinderbox/build-seamonkey-util.pm, so the tinderbox code will need to be retagged (should be doing this anyway)
** <tt>cvs update</tt> tinderbox to that tag on the three builders
** for each platform that will use PGO (expecting Windows & Linux)
*** bring over the <tt>mk_add_options PROFILE_GEN_SCRIPT...</tt> line from the nightly mozconfig to the release one, and install the script
*** bring over <tt>$ProfiledBuild = 1;</tt> from nightly tinder-config.pl to release one
*** compare RAM allocated to nightly and release VM's
=Build Engineers=
=Build Engineers=
rhelmer
rhelmer, joduinn


=Bonsai queries=
=Bonsai queries=
Line 16: Line 6:


Last checkins:
Last checkins:
* RC1 2008-??-?? ??:??:??
* cvs [http://bonsai.mozilla.org/cvsquery.cgi?treeid=default&module=all&branch=HEAD&branchtype=match&dir=&file=&filetype=match&who=&whotype=match&sortby=Date&hours=2&date=explicit&mindate=2008-02-26+23%3A59+PST&maxdate=2008-03-03+11%3A16+PST&cvsroot=%2Fcvsroot 2008-03-03 11:15 PST ]
* l10n [http://bonsai-l10n.mozilla.org/cvsquery.cgi?treeid=default&module=all&branch=HEAD&branchtype=match&dir=&file=&filetype=match&who=&whotype=match&sortby=Date&hours=2&date=explicit&mindate=2008-02-26+23%3A59+PST&maxdate=2008-03-03+5%3A00+PST&cvsroot=%2Fl10n 2008-03-03 05:00 PST ]


=Bugs=
=Bugs=


Filed version bump bug in [https://bugzilla.mozilla.org/show_bug.cgi?id=409880 bug 409880]
Filed version bump bug in [https://bugzilla.mozilla.org/show_bug.cgi?id=418926  bug 418926 ]


=Tags=
=Tags=
Line 29: Line 20:
| style="background:#efefef" | '''Pull date'''
| style="background:#efefef" | '''Pull date'''
|-
|-
| rowspan="5" | cvsroot/mozilla
| rowspan="3" | cvsroot/mozilla
| GECKO19b3_20080205_RELBRANCH
| GECKO19b4_20080304_RELBRANCH
| HEAD @ 2008-02-04 17:20 PST
| HEAD @ 2008-03-03 11:15 PST  
|-
|-
| FIREFOX_3_0b3_RC1
| FIREFOX_3_0b4_RC1
| HEAD @ 2008-02-04 17:20 PST
| GECKO19b4_20080304_RELBRANCH @ 2008-03-03 16:19 PST  
|-
 
| FIREFOX_3_0b3_RC2
| HEAD @ 2008-02-04 17:20 PST
|-
| FIREFOX_3_0b3_RC3
| HEAD @ 2008-02-04 17:20 PST
|-
| FIREFOX_3_0b3_RELEASE
| HEAD @ 2008-02-04 17:20 PST
|-
| rowspan="5" | l10n/l10n
| GECKO19b3_20080205_RELBRANCH
| HEAD @ 2008-02-04 17:20 PST
|-
|-
| FIREFOX_3_0b3_RC1
| FIREFOX_3_0b4_RELEASE
| HEAD @ 2008-02-04 17:20 PST
| GECKO19b4_20080304_RELBRANCH @ 2008-03-03 16:19 PST  
|-
|-
| FIREFOX_3_0b3_RC2
| rowspan="3" | l10n/l10n
| HEAD @ 2008-02-04 17:20 PST
| GECKO19b4_20080304_RELBRANCH
| HEAD @ 2008-03-03 05:00 PST
|-
|-
| FIREFOX_3_0b3_RC3
| FIREFOX_3_0b4_RC1
| HEAD @ 2008-02-06 03:30 PST
| GECKO19b4_20080304_RELBRANCH @ 2008-03-03 16:27 PST  
|-
|-
| FIREFOX_3_0b3_RELEASE
| FIREFOX_3_0b4_RELEASE
| HEAD @ 2008-02-06 03:30 PST
| GECKO19b4_20080304_RELBRANCH @ 2008-03-03 16:27 PST  
|-
|-
|}
|}
Line 94: Line 74:


==RC1==
==RC1==
Bootstrap tag: RELEASE_AUTOMATION_M7_1
Bootstrap tag: RELEASE_AUTOMATION_M7_2


Setup before starting:
Setup before starting:
 
* [http://developer.mozilla.org/en/docs/Building_with_Profile-Guided_Optimization Profile Guided Optimisation]
* verify nightly and release MOZCONFIG are in sync. Update release MOZCONFIG if needed.
** need at least rev1.383 of tinderbox/build-seamonkey-util.pm when tagging tinderbox
** for the Windows slave
*** bring over the <tt>mk_add_options PROFILE_GEN_SCRIPT...</tt> line from the nightly mozconfig to the release one, and the tweaks to CFLAGS & CXXFLAGS
*** Also pick up the new jemalloc line and removal of WIN32_REDIST_DIR
*** bring over <tt>$ProfiledBuild = 1;</tt> from nightly tinder-config.pl to release one
*** Bump fx-win32-1.9-slave2 up to 2GB of RAM, to match nightly VM
* used existing version bump bug to attached diff for bootstrap.cfg and master.cfg
* used existing version bump bug to attached diff for bootstrap.cfg and master.cfg
** make sure /builds/buildbot/trunk-automation-master/bootstrap.cfg is updated
** restarted master
** restarted master
* compare nightly and release mozconfig/tinder-config.pl files, merge any desired changes
* compare nightly and release mozconfig/tinder-config.pl files, merge any desired changes
* On fx-mac-1.9-slave2, fx-win32-1.9-slave2, fx-linux-1.9-slave2
* On fx-mac-1.9-slave2, fx-win32-1.9-slave2, fx-linux-1.9-slave2
** update tinderbox to RELEASE_AUTOMATION_M7_1
** '''REMEMBER to use VNC when restarting mac slave'''
** verify that both mac and linux have "xhost +" since last reboot
** update tinderbox to RELEASE_AUTOMATION_M7_2
*** FIXME update should be automatic - [https://bugzilla.mozilla.org/show_bug.cgi?id=397554 bug 397554]
*** FIXME update should be automatic - [https://bugzilla.mozilla.org/show_bug.cgi?id=397554 bug 397554]
*** cd /builds/tinderbox/mozilla/tools/tinderbox
*** cvs up -r RELEASE_AUTOMATION_M7_2
** remove the contents of /builds/verify/*
** remove the contents of /builds/verify/*
*** FIXME should be automatically backed up or removed - [https://bugzilla.mozilla.org/show_bug.cgi?id=413178 bug 413178]
*** FIXME should be automatically backed up or removed - [https://bugzilla.mozilla.org/show_bug.cgi?id=413178 bug 413178]
** restart buildbot  
** restart buildbot  
* On production-1.9-master
* On production-1.9-master
* remove the contents of /builds/updates/
** remove the contents of /builds/updates/
* remove private and public areas
** remove private and public areas
** rm -rf /data/cltbld/firefox-*
*** rm -rf /data/cltbld/firefox-*
*** FIXME should not be needed - [https://bugzilla.mozilla.org/show_bug.cgi?id=415970 bug 415970]
**** FIXME should not be needed - [https://bugzilla.mozilla.org/show_bug.cgi?id=415970 bug 415970]
** rm -rf /builds/tags/* - did not do this, plenty of space
*** rm -rf /builds/tags/* - did not do this, plenty of space
*** FIXME should not be needed - [https://bugzilla.mozilla.org/show_bug.cgi?id=415970 bug 415970]
**** FIXME should not be needed - [https://bugzilla.mozilla.org/show_bug.cgi?id=415970 bug 415970]
** in /home/ftp/pub/firefox/nightly/, remove all except the 3.x-candidates directory for the previous release  
*** in /home/ftp/pub/firefox/nightly/, remove all except the 3.x-candidates directory for the previous release  
*** rm -rf /home/ftp/pub/firefox/nightly/2007*
**** rm -rf /home/ftp/pub/firefox/nightly/2007*
*** FIXME should not be needed - [https://bugzilla.mozilla.org/show_bug.cgi?id=415970 bug 415970]
**** FIXME should not be needed - [https://bugzilla.mozilla.org/show_bug.cgi?id=415970 bug 415970]
* af was not tagged, so had to checkout a tagged locale so CVS would cache it
** cvs co -r FIREFOX_3_0b4_RELEASE l10n/de


* kick off buildbot (run as cltbld):
* kick off buildbot (run as cltbld):
  buildbot sendchange --username=joduinn --master=localhost:9989 -m "Firefox 3.0 Beta 4 RC1" release
  buildbot sendchange --username=joduinn --master=localhost:9989 -m "Firefox 3.0 Beta 4 RC1" release
====Prestage====
====Prestage====
* had to mkdir /home/cltbld/trunk-automation/prestage/build, not created automatically
* prestage removed /home/ftp/pub/firefox/nightly/3.0b3-candidates/ from the master, and that's needed later for l10n_verify. Manually synced the bits back from stage.m.o; patch on bug to fix this permanently
** FIXME buildbot bug?
 
====Tag====
====Tag====
* noticed that linux VMs/slaves are running one hour ahead of Mac, Win32 slaves. This doesnt make sense because we thought they were all ntp-enabled.
* nothing to do


====Source====
====Source====
* had to chmod g-s /home/ftp/ again, looks like it's coming from prestage step
* nothing to do
** rhelmer found we're setting it right here! http://mxr.mozilla.org/mozilla/source/tools/buildbot-configs/automation/production-1.9/master.cfg#102
* VERIFYFIX: confirmed that [https://bugzilla.mozilla.org/show_bug.cgi?id=409880 bug 409880] was truely fixed.
*** FIXED in [https://bugzilla.mozilla.org/show_bug.cgi?id=409880 bug 409880]


====Build====
====Build====
* win32 failed unit tests
* Mac was started from console not VNC, yielding "CFMessagePortCreateLocal failed" (security context problem in AliveTest).
** Config::General was not installed
** Removed all tags from tinder/mozconfigs, restarted from VNC.
*** installed it
**  FIXME slaves should come up usable on boot - bug 417887
** /builds/logs did not exist
 
*** created it
* Linux failed AliveTest (could not connect to X server)
** this was an oversight when MozillaBuild was upgraded
** Linux was rebooted and "xhost +" was not run.
** Removed all tags from tinder/mozconfigs, restarted from VNC.
** FIXME slaves should come up usable on boot - bug 417887
 
* SyncToStaging() failed
** [https://bugzilla.mozilla.org/show_bug.cgi?id=420824 bug 420824] filed for immediate relief
** commented out everything up to repack and resent change
** FIXME related to [https://bugzilla.mozilla.org/show_bug.cgi?id=398148 bug 398148]?
 
* (all platforms) Repack had leftover locale build override from 3.0b3 RC3
** backed out in [https://bugzilla.mozilla.org/show_bug.cgi?id=418926 bug 418926]
 
* Linux Repack did depend build, kernel was updated and bootstrap.cfg was not
** manually clobbered last-built and re-ran linux


* mac failed repack
* (all platforms) Repack was re-run without cleaning logs, so had to copy hourly directory manually to candidates dir and re-push to staging.
** af was not tagged, so had to checkout a tagged locale so CVS would cache it
*** FIXME should just build all locales [https://bugzilla.mozilla.org/show_bug.cgi?id=415895 bug 415895]


* linux failed build
* Linux Repack - rebuilt uk, zh-TW, zh-CN, because old builddir (derived from kernel version) was removed during run, which disrupted the previous run.
** AliveTest could not run, X server was refusing connections
*** ran "xhost +"
**** FIXME slaves should come up usable on boot - [https://bugzilla.mozilla.org/show_bug.cgi?id=417887 bug 417887]


====Sign====
====Sign====
[https://intranet.mozilla.org/Build:Signing Signing doc]
[https://intranet.mozilla.org/Build:Signing Signing doc]
* Build declared rc1 broken because of missed updates to mozconfig files as part of {{bug|#407794}}. Manually bringing release and nightly mozconfig files back in sync, and starting rc2.


====L10nVerify====
====L10nVerify====


====Updates to Test Channel====
====Updates to Test Channel====
* had to add DisableCompleteJump to rc list and re-push AUS config
 
* fallout from [https://bugzilla.mozilla.org/show_bug.cgi?id=410006 bug 410006] (prettyVersion feature); AUS dir "3 Beta 3" is created, not "3.0b3".
** workaround [https://bugzilla.mozilla.org/attachment.cgi?id=301534&action=edit patch] in [https://bugzilla.mozilla.org/show_bug.cgi?id=418926 bug 418926], re-gen and re-push AUS config
** FIXME permanent fix in [https://bugzilla.mozilla.org/show_bug.cgi?id=420947 bug 420947]


* verify beta snippets versus betatest snippets
* verify beta snippets versus betatest snippets
** only URL (download vs. stage) and appv ("3 Beta 3" vs. "3 Beta 3rc3")
** only URL (download vs. stage) and appv ("3 Beta 3" vs. "3 Beta 3rc3")
** FIXME automate this in [https://bugzilla.mozilla.org/show_bug.cgi?id=409449 bug 409449]
** FIXME automate this in [https://bugzilla.mozilla.org/show_bug.cgi?id=409449 bug 409449]
  cd /opt/aus2/snippets/staging/20080207-Firefox-3.0b3/
  cd /opt/aus2/snippets/staging/20080304-Firefox-3.0b4/
  find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/betatest/; system("diff -r -u $_ ../20080207-Firefox-3.0b3-test/$a");' 2>&1 | tee /tmp/beta.log
  find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/betatest/; system("diff -r -u $_ ../20080304-Firefox-3.0b4-test/$a");' 2>&1 | tee /tmp/beta.log
 
* verify beta snippets versus releasetest snippets
** FIXME automate this in [https://bugzilla.mozilla.org/show_bug.cgi?id=409449 bug 409449]
cd /opt/aus2/snippets/staging/20080304-Firefox-3.0b4/
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/releasetest/; system("diff -r -u $_ ../20080304-Firefox-3.0b4-test/$a");'
 
* FIXME: PatcherConfig made beta like betatest not releasetest. Posted patch in [https://bugzilla.mozilla.org/show_bug.cgi?id=418926 bug 418926], recreated/repushed snippets
 
* ensure that new test snippets are the same as old
** no changes expected
diff -r 20080304-Firefox-3.0b4-test.old 20080304-Firefox-3.0b4-test
 
* verify beta snippets versus betatest snippets
** only URL (download vs. stage) and appv ("3 Beta 4" vs. "3 Beta 4rc1")
** FIXME automate this in [https://bugzilla.mozilla.org/show_bug.cgi?id=409449 bug 409449]
cd /opt/aus2/snippets/staging/20080304-Firefox-3.0b4/
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/betatest/; system("diff -r -u $_ ../20080304-Firefox-3.0b4-test/$a");' 2>&1 | tee /tmp/beta.log


* verify beta snippets versus releasetest snippets
* verify beta snippets versus releasetest snippets
** no changes expected
** no changes expected
** FIXME automate this in [https://bugzilla.mozilla.org/show_bug.cgi?id=409449 bug 409449]
** FIXME automate this in [https://bugzilla.mozilla.org/show_bug.cgi?id=409449 bug 409449]
  cd /opt/aus2/snippets/staging/20080207-Firefox-3.0b3/
  cd /opt/aus2/snippets/staging/20080304-Firefox-3.0b4/
  find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/releasetest/; system("diff -r -u $_ ../20080207-Firefox-3.0b3-test/$a");'
  find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/releasetest/; system("diff -r -u $_ ../20080304-Firefox-3.0b4-test/$a");'
 
====update Verify====
 
* results uploaded to [https://build.mozilla.org/logs/Firefox_3.0b4_RC1/ https://build.mozilla.org/logs/Firefox_3.0b4_RC1/]
** VERIFYFIX: need to manually remove bump, updateverify config bump only works for rc1 [https://bugzilla.mozilla.org/show_bug.cgi?id=415261 bug 415261]
 
====Stage====
* ran out of disk space. Nagios warned us, but filled up remaining space faster then we could make more space.
** FIXME: change nagios threshold to be higher, so we have more time to react.
**  FIXME should move updates off, and/or add more disk
 
====Sign Installers====
 
* '''NOTE: this was not done: "pull stage:/data/cltbld/firefox-3.0b2/batch1/stage-signed/" '''
* Done manually using these installer-signing-instructions [https://intranet.mozilla.org/Build:Unified_Release_Process#Sign_builds here]
* push signed bits back to same location on stage
* complete stage-merged:
# on stage
cd /data/cltbld/firefox-3.0b4/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/
* Create MD5 and SHA1 checksum files
# on stage
cd /data/cltbld/firefox-3.0b4/stage-merged/
~/bin/checksum-files .
* Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:firefox .
chmod 644 *SUMS
 
====Update Bouncer====
* Manually add entries to download.mozilla.org
 
====Final Verification====
* manually verified files under /data/cltbld/firefox-3.0b4/stage-merged/
** looked ok.
* VERIFYFIX verified that filename problems from 3.b2 have been fixed. {{bug|409394}}
** verified ok.
 
====Push to mirrors====
* push the stage-merged directory to the releases area:
# on stage
rsync -av /data/cltbld/firefox-3.0b4/stage-merged/ /home/ftp/pub/firefox/releases/3.0b4/
 
* '''rc1 stopped because of {{bug|421528}}
 
== RC2  ==
 
Bootstrap tag: RELEASE_AUTOMATION_M7_2
 
Setup before starting:
* [http://developer.mozilla.org/en/docs/Building_with_Profile-Guided_Optimization Profile Guided Optimisation]
** need at least rev1.383 of tinderbox/build-seamonkey-util.pm when tagging tinderbox
** for the Windows slave
*** bring over the <tt>mk_add_options PROFILE_GEN_SCRIPT...</tt> line from the nightly mozconfig to the release one, and the tweaks to CFLAGS & CXXFLAGS
*** Also pick up the new jemalloc line and removal of WIN32_REDIST_DIR
*** bring over <tt>$ProfiledBuild = 1;</tt> from nightly tinder-config.pl to release one
*** Bump fx-win32-1.9-slave2 up to 2GB of RAM, to match nightly VM
* used existing version bump bug to attached diff for bootstrap.cfg and master.cfg
** make sure /builds/buildbot/trunk-automation-master/bootstrap.cfg is updated
** restarted master
* compare nightly and release mozconfig/tinder-config.pl files, merge any desired changes
* On fx-mac-1.9-slave2, fx-win32-1.9-slave2, fx-linux-1.9-slave2
** '''REMEMBER to use VNC when restarting mac slave'''
** verify that both mac and linux have "xhost +" since last reboot
** update tinderbox to RELEASE_AUTOMATION_M7_2
*** FIXME update should be automatic - [https://bugzilla.mozilla.org/show_bug.cgi?id=397554 bug 397554]
*** cd /builds/tinderbox/mozilla/tools/tinderbox
*** cvs up -r RELEASE_AUTOMATION_M7_2
** remove the contents of /builds/verify/*
*** FIXME should be automatically backed up or removed - [https://bugzilla.mozilla.org/show_bug.cgi?id=413178 bug 413178]
** restart buildbot
* On production-1.9-master
** remove the contents of /builds/updates/
** remove private and public areas
*** rm -rf /data/cltbld/firefox-*
**** FIXME should not be needed - [https://bugzilla.mozilla.org/show_bug.cgi?id=415970 bug 415970]
*** rm -rf /builds/tags/* - did not do this, plenty of space
**** FIXME should not be needed - [https://bugzilla.mozilla.org/show_bug.cgi?id=415970 bug 415970]
*** in /home/ftp/pub/firefox/nightly/, remove all except the 3.x-candidates directory for the previous release
**** rm -rf /home/ftp/pub/firefox/nightly/2007*
**** FIXME should not be needed - [https://bugzilla.mozilla.org/show_bug.cgi?id=415970 bug 415970]
 
* Win32 Tinderbox was missing PGO support. Tinderbox must be copied on MSYS, symlinks do not work. Copied /e/builds/tinderbox/mozilla/tools/tinderbox/* to:
/e/fx19rel/
/e/fx19l10nrel/
/e/builds/tinderbox/fx19rel/
/e/builds/tinderbox/fx19l10nrel/
 
* disabled linux/mac builders in master.cfg
 
* kick off buildbot (run as cltbld):
buildbot sendchange --username=joduinn --master=localhost:9989 -m "Firefox 3.0 Beta 4 RC2" release
Need to update master.cfg and restart/reconfig the Buildbot master for [https://bugzilla.mozilla.org/attachment.cgi?id=307141 a fix]
 
====Prestage====
* automated, nothing to do
 
====Tag====
* automated, nothing to do
 
====Source====
* automated, nothing to do
 
====Build====
 
* only rebuilding win32, so symlinked linux/mac on stage:
cd /home/ftp/pub/firefox/nightly/3.0b4-candidates
ln -s `pwd`/rc1/*linux* `pwd`/rc2/
ln -s `pwd`/rc1/*mac* `pwd`/rc2/
# remove win32 ja-JP-mac
rm -v rc2/firefox-3.0b4.ja-JP-mac.win32*
# remove partials
rm -v rc2/*partial*
# on production-1.8-master, sync from stage
rsync -av cltbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.0b4-candidates/ /home/ftp/pub/firefox/nightly/3.0b4-candidates/
 
* turns out absolute links no longer work, at least unless you use the new root (/pub/mozilla.org) - {{bug|421783}}. Fix it up with
cd /home/ftp/pub/firefox/nightly/3.0b4-candidates/rc2/
find . -type l | xargs rm -v
ln -sv ../rc1/*linux* .
ln -sv ../rc1/*mac* .
 
====Sign====
[https://intranet.mozilla.org/Build:Signing Signing doc]
 
====L10nVerify====
 
 
 
====Updates to Test Channel====
 
* verify beta snippets versus betatest snippets
** only URL (download vs. stage) and appv ("3 Beta 3" vs. "3 Beta 3rc3")
** FIXME automate this in [https://bugzilla.mozilla.org/show_bug.cgi?id=409449 bug 409449]
cd /opt/aus2/snippets/staging/20080308-Firefox-3.0b4/
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/betatest/; system("diff -r -u $_ ../20080308-Firefox-3.0b4-test/$a");' 2>&1 | tee /tmp/beta.log


* verify failed, PatcherConfig made beta like betatest not releasetest. Posted patch in [https://bugzilla.mozilla.org/show_bug.cgi?id=409880 bug 409880], recreated/repushed snippets
* verify beta snippets versus releasetest snippets
** FIXME automate this in [https://bugzilla.mozilla.org/show_bug.cgi?id=409449 bug 409449]
cd /opt/aus2/snippets/staging/20080308-Firefox-3.0b4/
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/releasetest/; system("diff -r -u $_ ../20080308-Firefox-3.0b4-test/$a");'
 
* FIXME: PatcherConfig made beta like betatest not releasetest. Posted patch in [https://bugzilla.mozilla.org/show_bug.cgi?id=418926 bug 418926], recreated/repushed snippets


* ensure that new test snippets are the same as old
* ensure that new test snippets are the same as old
** no changes expected
** no changes expected
  diff -r 20080207-Firefox-3.0b3-test 20080208-Firefox-3.0b3-test
  diff -r 20080308-Firefox-3.0b4-test 20080308-Firefox-3.0b4-test


* verify beta snippets versus betatest snippets
* verify beta snippets versus betatest snippets
** only URL (download vs. stage) and appv ("3 Beta 3" vs. "3 Beta 3rc3")
** only URL (download vs. stage) and appv ("3 Beta 4" vs. "3 Beta 4rc1")
** FIXME automate this in [https://bugzilla.mozilla.org/show_bug.cgi?id=409449 bug 409449]
** FIXME automate this in [https://bugzilla.mozilla.org/show_bug.cgi?id=409449 bug 409449]
  cd /opt/aus2/snippets/staging/20080208-Firefox-3.0b3/
  cd /opt/aus2/snippets/staging/20080308-Firefox-3.0b4/
  find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/betatest/; system("diff -r -u $_ ../20080208-Firefox-3.0b3-test/$a");' 2>&1 | tee /tmp/beta.log
  find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/betatest/; system("diff -r -u $_ ../20080308-Firefox-3.0b4-test/$a");' 2>&1 | tee /tmp/beta.log


* verify beta snippets versus releasetest snippets
* verify beta snippets versus releasetest snippets
** no changes expected
** no changes expected
** FIXME automate this in [https://bugzilla.mozilla.org/show_bug.cgi?id=409449 bug 409449]
** FIXME automate this in [https://bugzilla.mozilla.org/show_bug.cgi?id=409449 bug 409449]
  cd /opt/aus2/snippets/staging/20080208-Firefox-3.0b3/
  cd /opt/aus2/snippets/staging/20080308-Firefox-3.0b4/
  find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/releasetest/; system("diff -r -u $_ ../20080208-Firefox-3.0b3-test/$a");'
  find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/releasetest/; system("diff -r -u $_ ../20080308-Firefox-3.0b4-test/$a");'


====update Verify====
====update Verify====


* results uploaded to [https://build.mozilla.org/logs/Firefox_3.0b3_RC3/ https://build.mozilla.org/logs/Firefox_3.0b3_RC3/]
* win32 FAILed due to:
** FIXME need to manually remove bump, updateverify config bump only works for rc1 [https://bugzilla.mozilla.org/show_bug.cgi?id=415261 bug 415261]
** binary diffs in .chk files; expected failure
** TODO link bug
** missing el and es-AR locales in 3.0b3
* linux FAILed due to missing el and es-AR locales in 3.0b3
* mac FAILed due to missing el and es-AR locales in 3.0b3


====Stage====
====Stage====
* ran out of disk space; cleared out 3.0b3-candidates, nightly/2008* files and restarted
** FIXME: change nagios threshold to be higher, so we have more time to react.
** FIXME should move updates off, and/or add more disk
====Sign Installers====
====Sign Installers====


Done manually.
* '''NOTE: this was not done: "pull stage:/data/cltbld/firefox-3.0b2/batch1/stage-signed/" '''
* pull stage:/data/cltbld/firefox-3.0b2/batch1/stage-signed/
* Done manually using these installer-signing-instructions [https://intranet.mozilla.org/Build:Unified_Release_Process#Sign_builds here]
using these installer-signing-instructions [https://intranet.mozilla.org/Build:Unified_Release_Process#Sign_builds here]
* push signed bits back to same location on stage
* push signed bits back to same location on stage
* complete stage-merged:
* complete stage-merged:
  # on stage
  # on stage
  cd /data/cltbld/firefox-3.0b3/
  cd /data/cltbld/firefox-3.0b4/
  rsync -av batch1/mar/ stage-merged/
  rsync -av batch1/mar/ stage-merged/
  rsync -av batch1/stage-signed/ stage-merged/
  rsync -av batch1/stage-signed/ stage-merged/
* Create MD5 and SHA1 checksum files
* Create MD5 and SHA1 checksum files
  # on stage
  # on stage
  cd /data/cltbld/firefox-3.0b3/stage-merged/
  cd /data/cltbld/firefox-3.0b4/stage-merged/
  ~/bin/checksum-files .
  ~/bin/checksum-files .
* Fix permissions & ownership (on the two SUM files, and the detached sigs)
* Fix permissions & ownership (on the two SUM files, and the detached sigs)
Line 214: Line 381:


====Update Bouncer====
====Update Bouncer====
* Manually add entries to download.mozilla.org
* already done for rc1


====Final Verification====
====Final Verification====
* manually verified files under /data/cltbld/firefox-3.0b3/stage-merged/ looked ok.  
* manually verified files under /data/cltbld/firefox-3.0b4/stage-merged/  
* verified that filename problems from 3.b2 have been fixed. {{bug|409394}}
** looked ok.  
* VERIFYFIX verified that filename problems from 3.b2 have been fixed. {{bug|409394}}
** verified ok.


====Push to mirrors====
====Push to mirrors====
* push the stage-merged directory to the releases area:
* push the stage-merged directory to the releases area:
  # on stage
  # on stage
  rsync -av /data/cltbld/firefox-3.0b3/stage-merged/ /home/ftp/pub/firefox/releases/3.0b3/
  rsync -av /data/cltbld/firefox-3.0b4/stage-merged/ /home/ftp/pub/firefox/releases/3.0b4/
* TODO: note that in /data/cltbld, the "firefox-3.0b4" directory should probably be "firefox-3.0b4rc2". This would make it easier to confirm changes between RCs. This is done for some releases not all; should we always do this?
 


====Updates to Beta Channel====
====Updates to Beta Channel====
 
'''In case you need to cancel/abort updates, the quickest way to do this is [https://intranet.mozilla.org/Build:Farm:Tinderbox_Maintenance#Starting.2FStopping_Tinderbox_on_win32]'''
* on Mac build machines, run quick updateverify test for releastest channel
* on Mac build machines, run quick updateverify test for releastest channel
  cd /builds/verify/firefox-3.0b3/updates
  cd /builds/verify/firefox-3.0b4/updates
  cat moz19-firefox-*.cfg > releasetest.cfg
  cat moz19-firefox-*.cfg > releasetest.cfg
  sed -i.bak 's/betatest/releasetest/' releasetest.cfg
  sed -i.bak 's/betatest/releasetest/' releasetest.cfg
Line 237: Line 408:
  grep 'HTTP' releasetest.log  | grep -v Found | grep -v 200
  grep 'HTTP' releasetest.log  | grep -v Found | grep -v 200


* Ensure that releasetest channel contents match beta channel contents
* On aus2-staging, ensure that releasetest channel contents match beta channel contents
  cd /opt/aus2/snippets/staging/20080208-Firefox-3.0b3
  cd /opt/aus2/snippets/staging/20080308-Firefox-3.0b4
  find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/releasetest/; system("diff -r -u $_ ../20080208-Firefox-3.0b3-test/$a");'  
  find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/releasetest/; system("diff -r -u $_ ../20080308-Firefox-3.0b4-test/$a");'  


No difs found - which is correct.
No difs found - which is correct.
Line 245: Line 416:
* Once QA and Website gives formal "go"
* Once QA and Website gives formal "go"
  # put snippets on beta
  # put snippets on beta
# login to aus2-staging
  $ sudo su - cltbld
  $ sudo su - cltbld
  $ cd /opt/aus2/snippets/staging
  $ cd /opt/aus2/snippets/staging
  $ ~/bin/pushsnip 20080208-Firefox-3.0b3
  $ ~/bin/pushsnip 20080308-Firefox-3.0b4


* on Mac build machines, run quick updateverify test for beta channel
* on Mac build machines, run quick updateverify test for beta channel
  cd /builds/verify/firefox-3.0b3/updates
  cd /builds/verify/firefox-3.0b4/updates
  cat moz19-firefox-*.cfg > beta.cfg
  cat moz19-firefox-*.cfg > beta.cfg
  sed -i.bak 's/betatest/beta/' beta.cfg
  sed -i.bak 's/betatest/beta/' beta.cfg
Line 259: Line 431:
  grep 'HTTP' releasetest.log  | grep -v Found | grep -v 200
  grep 'HTTP' releasetest.log  | grep -v Found | grep -v 200


=== Wall clock timings ===
== Wall clock timings ==
 
== Stopping live updates (just in case!) ==
If QA finds a problem with updates *after* we start pushing updates "live", the one-and-only-one way we know to stop updates from further spreading is by killing all updates!
 
See instructions at: https://intranet.mozilla.org/Build:Updates .

Latest revision as of 00:19, 21 March 2008

Build Engineers

rhelmer, joduinn

Bonsai queries

Was anything checked in after the "door closed"?

Last checkins:

Bugs

Filed version bump bug in bug 418926

Tags

Module Tag Pull date
cvsroot/mozilla GECKO19b4_20080304_RELBRANCH HEAD @ 2008-03-03 11:15 PST
FIREFOX_3_0b4_RC1 GECKO19b4_20080304_RELBRANCH @ 2008-03-03 16:19 PST
FIREFOX_3_0b4_RELEASE GECKO19b4_20080304_RELBRANCH @ 2008-03-03 16:19 PST
l10n/l10n GECKO19b4_20080304_RELBRANCH HEAD @ 2008-03-03 05:00 PST
FIREFOX_3_0b4_RC1 GECKO19b4_20080304_RELBRANCH @ 2008-03-03 16:27 PST
FIREFOX_3_0b4_RELEASE GECKO19b4_20080304_RELBRANCH @ 2008-03-03 16:27 PST

Build data

Type Build ID SHA1 Push date Build machine
[Windows installer] en-US:fx-win32-1.9-slave2 l10n:fx-win32-1.9-slave2
[Mac compressed] en-US:fx-mac-1.9-slave2 l10n:fx-mac-1.9-slave2
[Linux compressed] en-US:fx-linux-1.9-slave2 l10n:fx-linux-1.9-slave2

Notes

RC1

Bootstrap tag: RELEASE_AUTOMATION_M7_2

Setup before starting:

  • Profile Guided Optimisation
    • need at least rev1.383 of tinderbox/build-seamonkey-util.pm when tagging tinderbox
    • for the Windows slave
      • bring over the mk_add_options PROFILE_GEN_SCRIPT... line from the nightly mozconfig to the release one, and the tweaks to CFLAGS & CXXFLAGS
      • Also pick up the new jemalloc line and removal of WIN32_REDIST_DIR
      • bring over $ProfiledBuild = 1; from nightly tinder-config.pl to release one
      • Bump fx-win32-1.9-slave2 up to 2GB of RAM, to match nightly VM
  • used existing version bump bug to attached diff for bootstrap.cfg and master.cfg
    • make sure /builds/buildbot/trunk-automation-master/bootstrap.cfg is updated
    • restarted master
  • compare nightly and release mozconfig/tinder-config.pl files, merge any desired changes
  • On fx-mac-1.9-slave2, fx-win32-1.9-slave2, fx-linux-1.9-slave2
    • REMEMBER to use VNC when restarting mac slave
    • verify that both mac and linux have "xhost +" since last reboot
    • update tinderbox to RELEASE_AUTOMATION_M7_2
      • FIXME update should be automatic - bug 397554
      • cd /builds/tinderbox/mozilla/tools/tinderbox
      • cvs up -r RELEASE_AUTOMATION_M7_2
    • remove the contents of /builds/verify/*
      • FIXME should be automatically backed up or removed - bug 413178
    • restart buildbot
  • On production-1.9-master
    • remove the contents of /builds/updates/
    • remove private and public areas
      • rm -rf /data/cltbld/firefox-*
      • rm -rf /builds/tags/* - did not do this, plenty of space
      • in /home/ftp/pub/firefox/nightly/, remove all except the 3.x-candidates directory for the previous release
        • rm -rf /home/ftp/pub/firefox/nightly/2007*
        • FIXME should not be needed - bug 415970
  • af was not tagged, so had to checkout a tagged locale so CVS would cache it
    • cvs co -r FIREFOX_3_0b4_RELEASE l10n/de
  • kick off buildbot (run as cltbld):
buildbot sendchange --username=joduinn --master=localhost:9989 -m "Firefox 3.0 Beta 4 RC1" release

Prestage

  • prestage removed /home/ftp/pub/firefox/nightly/3.0b3-candidates/ from the master, and that's needed later for l10n_verify. Manually synced the bits back from stage.m.o; patch on bug to fix this permanently

Tag

  • nothing to do

Source

  • nothing to do
  • VERIFYFIX: confirmed that bug 409880 was truely fixed.

Build

  • Mac was started from console not VNC, yielding "CFMessagePortCreateLocal failed" (security context problem in AliveTest).
    • Removed all tags from tinder/mozconfigs, restarted from VNC.
    • FIXME slaves should come up usable on boot - bug 417887
  • Linux failed AliveTest (could not connect to X server)
    • Linux was rebooted and "xhost +" was not run.
    • Removed all tags from tinder/mozconfigs, restarted from VNC.
    • FIXME slaves should come up usable on boot - bug 417887
  • SyncToStaging() failed
    • bug 420824 filed for immediate relief
    • commented out everything up to repack and resent change
    • FIXME related to bug 398148?
  • (all platforms) Repack had leftover locale build override from 3.0b3 RC3
  • Linux Repack did depend build, kernel was updated and bootstrap.cfg was not
    • manually clobbered last-built and re-ran linux
  • (all platforms) Repack was re-run without cleaning logs, so had to copy hourly directory manually to candidates dir and re-push to staging.
  • Linux Repack - rebuilt uk, zh-TW, zh-CN, because old builddir (derived from kernel version) was removed during run, which disrupted the previous run.

Sign

Signing doc

L10nVerify

Updates to Test Channel

  • fallout from bug 410006 (prettyVersion feature); AUS dir "3 Beta 3" is created, not "3.0b3".
  • verify beta snippets versus betatest snippets
    • only URL (download vs. stage) and appv ("3 Beta 3" vs. "3 Beta 3rc3")
    • FIXME automate this in bug 409449
cd /opt/aus2/snippets/staging/20080304-Firefox-3.0b4/
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/betatest/; system("diff -r -u $_ ../20080304-Firefox-3.0b4-test/$a");' 2>&1 | tee /tmp/beta.log
  • verify beta snippets versus releasetest snippets 
cd /opt/aus2/snippets/staging/20080304-Firefox-3.0b4/
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/releasetest/; system("diff -r -u $_ ../20080304-Firefox-3.0b4-test/$a");'
  • FIXME: PatcherConfig made beta like betatest not releasetest. Posted patch in bug 418926, recreated/repushed snippets
  • ensure that new test snippets are the same as old
    • no changes expected
diff -r 20080304-Firefox-3.0b4-test.old 20080304-Firefox-3.0b4-test
  • verify beta snippets versus betatest snippets
    • only URL (download vs. stage) and appv ("3 Beta 4" vs. "3 Beta 4rc1")
    • FIXME automate this in bug 409449
cd /opt/aus2/snippets/staging/20080304-Firefox-3.0b4/
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/betatest/; system("diff -r -u $_ ../20080304-Firefox-3.0b4-test/$a");' 2>&1 | tee /tmp/beta.log
  • verify beta snippets versus releasetest snippets
    • no changes expected
    • FIXME automate this in bug 409449
cd /opt/aus2/snippets/staging/20080304-Firefox-3.0b4/
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/releasetest/; system("diff -r -u $_ ../20080304-Firefox-3.0b4-test/$a");'

update Verify

Stage

  • ran out of disk space. Nagios warned us, but filled up remaining space faster then we could make more space.
    • FIXME: change nagios threshold to be higher, so we have more time to react.
    • FIXME should move updates off, and/or add more disk

Sign Installers

  • NOTE: this was not done: "pull stage:/data/cltbld/firefox-3.0b2/batch1/stage-signed/"
  • Done manually using these installer-signing-instructions here
  • push signed bits back to same location on stage
  • complete stage-merged:
# on stage
cd /data/cltbld/firefox-3.0b4/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/
  • Create MD5 and SHA1 checksum files
# on stage
cd /data/cltbld/firefox-3.0b4/stage-merged/
~/bin/checksum-files .
  • Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:firefox .
chmod 644 *SUMS

Update Bouncer

  • Manually add entries to download.mozilla.org

Final Verification

  • manually verified files under /data/cltbld/firefox-3.0b4/stage-merged/
    • looked ok.
  • VERIFYFIX verified that filename problems from 3.b2 have been fixed. bug 409394
    • verified ok.

Push to mirrors

  • push the stage-merged directory to the releases area:
# on stage
rsync -av /data/cltbld/firefox-3.0b4/stage-merged/ /home/ftp/pub/firefox/releases/3.0b4/

RC2

Bootstrap tag: RELEASE_AUTOMATION_M7_2

Setup before starting:

  • Profile Guided Optimisation
    • need at least rev1.383 of tinderbox/build-seamonkey-util.pm when tagging tinderbox
    • for the Windows slave
      • bring over the mk_add_options PROFILE_GEN_SCRIPT... line from the nightly mozconfig to the release one, and the tweaks to CFLAGS & CXXFLAGS
      • Also pick up the new jemalloc line and removal of WIN32_REDIST_DIR
      • bring over $ProfiledBuild = 1; from nightly tinder-config.pl to release one
      • Bump fx-win32-1.9-slave2 up to 2GB of RAM, to match nightly VM
  • used existing version bump bug to attached diff for bootstrap.cfg and master.cfg
    • make sure /builds/buildbot/trunk-automation-master/bootstrap.cfg is updated
    • restarted master
  • compare nightly and release mozconfig/tinder-config.pl files, merge any desired changes
  • On fx-mac-1.9-slave2, fx-win32-1.9-slave2, fx-linux-1.9-slave2
    • REMEMBER to use VNC when restarting mac slave
    • verify that both mac and linux have "xhost +" since last reboot
    • update tinderbox to RELEASE_AUTOMATION_M7_2
      • FIXME update should be automatic - bug 397554
      • cd /builds/tinderbox/mozilla/tools/tinderbox
      • cvs up -r RELEASE_AUTOMATION_M7_2
    • remove the contents of /builds/verify/*
      • FIXME should be automatically backed up or removed - bug 413178
    • restart buildbot
  • On production-1.9-master
    • remove the contents of /builds/updates/
    • remove private and public areas
      • rm -rf /data/cltbld/firefox-*
      • rm -rf /builds/tags/* - did not do this, plenty of space
      • in /home/ftp/pub/firefox/nightly/, remove all except the 3.x-candidates directory for the previous release
        • rm -rf /home/ftp/pub/firefox/nightly/2007*
        • FIXME should not be needed - bug 415970
  • Win32 Tinderbox was missing PGO support. Tinderbox must be copied on MSYS, symlinks do not work. Copied /e/builds/tinderbox/mozilla/tools/tinderbox/* to:
/e/fx19rel/
/e/fx19l10nrel/
/e/builds/tinderbox/fx19rel/
/e/builds/tinderbox/fx19l10nrel/
  • disabled linux/mac builders in master.cfg
  • kick off buildbot (run as cltbld):
buildbot sendchange --username=joduinn --master=localhost:9989 -m "Firefox 3.0 Beta 4 RC2" release

Need to update master.cfg and restart/reconfig the Buildbot master for a fix

Prestage

  • automated, nothing to do

Tag

  • automated, nothing to do

Source

  • automated, nothing to do

Build

  • only rebuilding win32, so symlinked linux/mac on stage:
cd /home/ftp/pub/firefox/nightly/3.0b4-candidates
ln -s `pwd`/rc1/*linux* `pwd`/rc2/
ln -s `pwd`/rc1/*mac* `pwd`/rc2/
# remove win32 ja-JP-mac
rm -v rc2/firefox-3.0b4.ja-JP-mac.win32*
# remove partials
rm -v rc2/*partial*
# on production-1.8-master, sync from stage
rsync -av cltbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.0b4-candidates/ /home/ftp/pub/firefox/nightly/3.0b4-candidates/
  • turns out absolute links no longer work, at least unless you use the new root (/pub/mozilla.org) - bug 421783. Fix it up with
cd /home/ftp/pub/firefox/nightly/3.0b4-candidates/rc2/
find . -type l | xargs rm -v
ln -sv ../rc1/*linux* .
ln -sv ../rc1/*mac* .

Sign

Signing doc

L10nVerify

Updates to Test Channel

  • verify beta snippets versus betatest snippets
    • only URL (download vs. stage) and appv ("3 Beta 3" vs. "3 Beta 3rc3")
    • FIXME automate this in bug 409449
cd /opt/aus2/snippets/staging/20080308-Firefox-3.0b4/
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/betatest/; system("diff -r -u $_ ../20080308-Firefox-3.0b4-test/$a");' 2>&1 | tee /tmp/beta.log
  • verify beta snippets versus releasetest snippets 
cd /opt/aus2/snippets/staging/20080308-Firefox-3.0b4/
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/releasetest/; system("diff -r -u $_ ../20080308-Firefox-3.0b4-test/$a");'
  • FIXME: PatcherConfig made beta like betatest not releasetest. Posted patch in bug 418926, recreated/repushed snippets
  • ensure that new test snippets are the same as old
    • no changes expected
diff -r 20080308-Firefox-3.0b4-test 20080308-Firefox-3.0b4-test
  • verify beta snippets versus betatest snippets
    • only URL (download vs. stage) and appv ("3 Beta 4" vs. "3 Beta 4rc1")
    • FIXME automate this in bug 409449
cd /opt/aus2/snippets/staging/20080308-Firefox-3.0b4/
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/betatest/; system("diff -r -u $_ ../20080308-Firefox-3.0b4-test/$a");' 2>&1 | tee /tmp/beta.log
  • verify beta snippets versus releasetest snippets
    • no changes expected
    • FIXME automate this in bug 409449
cd /opt/aus2/snippets/staging/20080308-Firefox-3.0b4/
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/releasetest/; system("diff -r -u $_ ../20080308-Firefox-3.0b4-test/$a");'

update Verify

  • win32 FAILed due to:
    • binary diffs in .chk files; expected failure
    • TODO link bug
    • missing el and es-AR locales in 3.0b3
  • linux FAILed due to missing el and es-AR locales in 3.0b3
  • mac FAILed due to missing el and es-AR locales in 3.0b3

Stage

  • ran out of disk space; cleared out 3.0b3-candidates, nightly/2008* files and restarted
    • FIXME: change nagios threshold to be higher, so we have more time to react.
    • FIXME should move updates off, and/or add more disk

Sign Installers

  • NOTE: this was not done: "pull stage:/data/cltbld/firefox-3.0b2/batch1/stage-signed/"
  • Done manually using these installer-signing-instructions here
  • push signed bits back to same location on stage
  • complete stage-merged:
# on stage
cd /data/cltbld/firefox-3.0b4/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/
  • Create MD5 and SHA1 checksum files
# on stage
cd /data/cltbld/firefox-3.0b4/stage-merged/
~/bin/checksum-files .
  • Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:firefox .
chmod 644 *SUMS

Update Bouncer

  • already done for rc1

Final Verification

  • manually verified files under /data/cltbld/firefox-3.0b4/stage-merged/
    • looked ok.
  • VERIFYFIX verified that filename problems from 3.b2 have been fixed. bug 409394
    • verified ok.

Push to mirrors

  • push the stage-merged directory to the releases area:
# on stage
rsync -av /data/cltbld/firefox-3.0b4/stage-merged/ /home/ftp/pub/firefox/releases/3.0b4/
  • TODO: note that in /data/cltbld, the "firefox-3.0b4" directory should probably be "firefox-3.0b4rc2". This would make it easier to confirm changes between RCs. This is done for some releases not all; should we always do this?


Updates to Beta Channel

In case you need to cancel/abort updates, the quickest way to do this is [1]

  • on Mac build machines, run quick updateverify test for releastest channel
cd /builds/verify/firefox-3.0b4/updates
cat moz19-firefox-*.cfg > releasetest.cfg
sed -i.bak 's/betatest/releasetest/' releasetest.cfg
./verify.sh -t releasetest.cfg 2>&1 | tee releasetest.log
# upload result to build.m.o/logs

Should be all HTTP/200 OK results e.g.: 

grep 'HTTP' releasetest.log  | grep -v Found | grep -v 200
  • On aus2-staging, ensure that releasetest channel contents match beta channel contents
cd /opt/aus2/snippets/staging/20080308-Firefox-3.0b4
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/releasetest/; system("diff -r -u $_ ../20080308-Firefox-3.0b4-test/$a");'

No difs found - which is correct.

  • Once QA and Website gives formal "go"
# put snippets on beta
# login to aus2-staging
$ sudo su - cltbld
$ cd /opt/aus2/snippets/staging
$ ~/bin/pushsnip 20080308-Firefox-3.0b4
  • on Mac build machines, run quick updateverify test for beta channel
cd /builds/verify/firefox-3.0b4/updates
cat moz19-firefox-*.cfg > beta.cfg
sed -i.bak 's/betatest/beta/' beta.cfg
./verify.sh -t beta.cfg 2>&1 | tee beta.log
# upload result to build.m.o/logs

Should be all HTTP/200 OK results e.g.: 

grep 'HTTP' releasetest.log  | grep -v Found | grep -v 200

Wall clock timings

Stopping live updates (just in case!)

If QA finds a problem with updates *after* we start pushing updates "live", the one-and-only-one way we know to stop updates from further spreading is by killing all updates!

See instructions at: https://intranet.mozilla.org/Build:Updates .

Retrieved from "https://wiki.allizom.org/index.php?title=Firefox_3.0b4:BuildNotes&oldid=87543"