MozillaWiki

Releases/Firefox 3.6.14/BuildNotes: Difference between revisions

Page Discussion

Releases/Firefox 3.6.14/BuildNotes (view source)

Revision as of 16:07, 2 March 2011

28,946 bytes added ,  2 March 2011
m
→‎Final Verification
 
(117 intermediate revisions by 6 users not shown)
Line 12: Line 12:
|-
|-
| 1
| 1
| FIREFOX_3_6_14_BUILD1 FIREFOX_3_6_14_RELEASE
| FIREFOX_3_6_14_BUILD1
| http://hg.mozilla.org/releases/mozilla-1.9.2/rev/69bfc730b3e9
| [http://hg.mozilla.org/releases/mozilla-1.9.2/rev/69bfc730b3e9 releases/mozilla-1.9.2/69bfc730b3e9]
|-
| 2
| FIREFOX_3_6_14_BUILD2
| [http://hg.mozilla.org/releases/mozilla-1.9.2/rev/77abb6fcd2b7 releases/mozilla-1.9.2/77abb6fcd2b7]
|-
| 3
| FIREFOX_3_6_14_BUILD3
| [http://hg.mozilla.org/releases/mozilla-1.9.2/rev/a73120a6aa93 releases/mozilla-1.9.2/a73120a6aa93]
|}
|}
= Notifications =
* We have to notify AV vendors once the signed bits are on candidates directory (for email address and template example, see {{bug|568884#c15}} )
* We have to give mirrors as much notice as possible before pushing to mirrors. This has been 24-72 hours, but has been up to a week in advance.  (for email address and template example, see {{bug|633657}} )
* We have to notify infra-all@ before "go to build" and before we ship. The more advance notice the better.
* We have to notify metrics once we release


= Bugs hit =
= Bugs hit =
* {{bug|616512}} - Linux repacks will always fail to upload if run before win32 build finishes
== To be hit on next release unless fixed ==
You won't hit these bugs if they are fixed first:
* <strike>{{bug|628795}} - en-US.xpi uploaded to the wrong place during 1.9.1/1.9.2 releases</strike>
* <strike>{{bug|616512}} - Linux repacks will always fail to upload if run before win32 build finishes</strike>
* <strike>{{bug|633961}} - pre push check fails</strike>
* [got backed out] {{bug|632399}} ctors steps
* <strike>{{bug|632412}} - signing regression for win32-EUBallot</strike>
* {{bug|632449}} - generate-candidate-build-updates.py fails when locales are removed between buildX and buildY of the same release
** this is only hit for a build2
* <strike>{{bug|633695}} - EUBallot should be renamed to EUballot (lower case)</strike>
 
== Sporadic/Not easily hitable ==
* {{bug|628390}} - macosx_repack_4/6 failed because post_upload failed for 'ko'
* {{bug|628386}} - win32_repack_5/6 failed to timeout on w32-ix-slave23 being slow
* {{bug|557268}} - "update_verify" dependent scheduler did not fire
* {{bug|633336}} - 2nd run of mac partner repacks fails because previous run did not remove mount point
* {{bug|629256}} - Fix snippets for some locales
* macosx_repack_2/6 died with SIGUSR1
* macosx_repack_2/6 died with SIGUSR1
** TODO check if it needs filing
* macosx_repack_4/6 failed because post_upload failed for 'ko'
** not sure how to file; maybe just document and close the bug?
* {{bug|628386}} - win32_repack_5/6 failed to timeout on w32-ix-slave23 being slow
* 2nd run of mac partner repacks fails because previous run did not remove mount point
** TODO file
* en-US gets posted
** TODO file a bug to backport fix for 1.9.1/1.9.2
** see {{bug|609878}}
** see {{bug|485860}}


=Build data=
=Build data=
Line 40: Line 60:
| rowspan="3" | 1
| rowspan="3" | 1
| Linux
| Linux
|  
| 20110121145245
| mv-moz2-linux-ix-slave11  
| mv-moz2-linux-ix-slave11  
|
| 38 mins, 28 secs
|-
|-
| Mac
| Mac
|
| 20110121145415
| bm-xserve16  
| bm-xserve16  
|
| 1 hrs, 39 mins, 47 secs
|-
|-
| Windows
| Windows
|  
| 20110121145327
| w32-ix-slave23  
| w32-ix-slave23  
| 4 hrs, 3 mins, 53 secs
|-
| rowspan="3" | 2
| Linux
| 20110207130507
| mv-moz2-linux-ix-slave11
| 39 mins, 27 secs
|-
| Mac
| 20110207130523
| bm-xserve16
| 1 hrs, 39 mins, 5 secs
|-
| Windows
| 20110207134212
| w32-ix-slave23 ('''slow IX''')
| 5 hrs, 11 mins, 35 secs
|-
| rowspan="3" | 3
| Linux
| 20110218124314
| moz2-linux-slave23
|  
|  
|-
| Mac
| 20110218124600
| bm-xserve16
|
|-
| Windows
| 20110218125750
| w32-ix-slave23 ('''slow IX''')
|
|}
|}


==Build 1==
=Build 1=
* Clobbered with "Any master/any release/any builder"
* Clobbered with "Any master/any release/any builder"
* tagged buildbotcustom, compare-locales and tools with FIREFOX_3_6_14_BUILD1 and FIREFOX_3_6_14_RELEASE
* tagged buildbotcustom, compare-locales and tools with FIREFOX_3_6_14_BUILD1 and FIREFOX_3_6_14_RELEASE
Line 70: Line 122:
=== Tag ===
=== Tag ===
No problems.
No problems.
=== Version bump ===
Please verify on your next release that {{bug|607372}} is fixed.
Once you do VERIFY the bug and remove this section.
=== Update Bouncer ===
Used "Force Build" to start the "release-mozilla-1.9.2-bouncer_submitter" builder.
All good.
=== Stage EU Ballot Builds ===
# ffxbld@stage
cd /home/ftp/pub/firefox/nightly/3.6.14-candidates/build1
mv -v partner-repacks/euballot/win32 win32-EUballot
Please remove this section once {{bug|614227}} is FIXED.


=== Source / XULRunner Source ===
=== Source / XULRunner Source ===
Line 76: Line 146:
=== Build ===
=== Build ===
* No problems
* No problems
=== Remove en-US.xpi ===
You want to remove the en-US.xpi before autosigning creates the MD5SUMS and the SHA1SUMS.
Please remove this section once {{bug|628795}} is fixed.
Check [[Releases/Firefox_3.6.14/BuildNotes#Signing|here]] if you don't do remove en-US.xpi on time and want to know how to recover.
=== Start autosigning ===
If you are expecting to sign during your workday you can start [https://intranet.mozilla.org/Build:CombinedSigning auto signing] to take care of the deliverables as they are uploaded to FTP.


=== L10n Repack ===
=== L10n Repack ===
Line 81: Line 161:
** Manually rebooted bm-xserve16
** Manually rebooted bm-xserve16
** Retriggered the builder with "rebuild"
** Retriggered the builder with "rebuild"
** TODO check if a bug is filed or if it is needed to be filed
** check [[Releases/Firefox_3.6.14/BuildNotes#Bugs_hit|bugs hit section]] for any filed bug
* macosx_repack_4/6 failed because post_upload failed for 'ko'
* macosx_repack_4/6 failed because post_upload failed for 'ko'
** TODO file bug
** used "macosx_standalone_repack" to re-trigger 'ko' as documented in [https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Documentation#L10n_Repack here]
** used "macosx_standalone_repack" to re-trigger 'ko' as documented in [https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Documentation#L10n_Repack here]
** check [[Releases/Firefox_3.6.14/BuildNotes#Bugs_hit|bugs hit section]] for any filed bug
* win32_repack_5/6 failed to timeout on w32-ix-slave23 being slow
* win32_repack_5/6 failed to timeout on w32-ix-slave23 being slow
** TODO file bug on bumping timeout
** TODO file bug on the slave being slow
** gracefully shutdown the slave and re-triggered the job
** gracefully shutdown the slave and re-triggered the job
** check [[Releases/Firefox_3.6.14/BuildNotes#Bugs_hit|bugs hit section]] for any filed bug


=== Partner Repack ===
=== Partner Repack ===
Line 101: Line 180:
  touch /pub/mozilla.org/firefox/nightly/3.6.14-candidates/build1//unsigned/partner-repacks/partner_build_linux'
  touch /pub/mozilla.org/firefox/nightly/3.6.14-candidates/build1//unsigned/partner-repacks/partner_build_linux'
  # after touching the autosign system can proceeed
  # after touching the autosign system can proceeed
** with the Mac repack that failed I re-triggered
* 2nd resubmitted Mac repack failed
*** FAILS: the reason is that the first job that failed did not run the cleanup job
** the reason is that the first job that failed did not run the cleanup job
*** TODO file bug with fix and explaining better
** check [[Releases/Firefox_3.6.14/BuildNotes#Bugs_hit|bugs hit section]] for any filed bug


* It is all fine now
* It is all fine now
Line 111: Line 190:


=== Unittests ===
=== Unittests ===
Linux:
* hit {{bug|565437}} timeout which aborted the run; I re-triggered it
Darwin:
* hit {{bug|501221}} or {{bug|502646}} timeout; nothing to be done
Windows:
* green


=== Signing ===
=== Signing ===
Currently running .
Followed [https://intranet.mozilla.org/Build:CombinedSigning CombinedSigning doc].


TODO I am afraid I did remove too late. After the "sign-release.py for firefox 3.6.14 starting" email was sent. Let's see.
I did remove the en-US.xpi file too late.
* bug filed under the [[Releases/Firefox_3.6.14/BuildNotes#Bugs_hit|bugs hit section]]
 
To recover I did the following:
  #ffxbld@stage-old.mozilla.org
  #ffxbld@stage-old.mozilla.org
  rm /home/ftp/pub/mozilla.org/firefox/nightly/3.6.14-candidates/build1/en-US.xpi
  rm /home/ftp/pub/mozilla.org/firefox/nightly/3.6.14-candidates/build1/en-US.xpi
On the keymaster:
cd ~/signing-work/firefox-3.6.4/signed_build1
rm en-US.xpi MD5SUMS* SHA1SUMS*
# I run the following command but by running each individually
# because I didn't know I could run it in one shot
# NOTE that "stage", "verify" and "fake-upload" are targets
# that happen after "sign2" is run
# http://hg.mozilla.org/build/tools/file/tip/release/signing/Makefile#l73
make checksum-files create-sigs verify-sigs quick-verify stage verify fake-upload PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG} USE_NEW=1
Then continued by uploading as it would normally be done.


No problems, followed [https://intranet.mozilla.org/Build:CombinedSigning CombinedSigning doc].
=== Notify AV vendors ===
* Notify vendors using instructions in {{bug|568884#c15}}


=== L10n Verify ===
=== L10n Verify ===
Many string changes across many platforms/locales. All seemed legitimate, didn't dig further.


=== Updates ===
=== Updates ===
* completion on Monday Jan. 24th @ 15:30pm PST.
* QA signed off @ 4:22pm PST
==== 3.6* -> 3.6.13 ====
==== 3.6* -> 3.6.13 ====
===== Generation =====
===== Generation =====
Got triggered after signing bits appeared on ftp.
No problems.


===== Update Verify =====
===== Update Verify =====
I had to "force build" the "update_verify" since reconfigures happened since the release was triggered. (added to [[Releases/Firefox_3.6.14/BuildNotes#Bugs_hit|bugs hit section]])
All went green.


=== Socorro product data updates ===
=== Socorro product data updates ===
Added '3.6.14' & '3.6.15pre' to crash-stats with [http://crash-stats.mozilla.com/admin/branch_data_sources this admin page].
== Going to beta day!! ==
Do this in the morning before the rel-driver gives you the GO.
#From a build slave you can reach aus2-staging
#cltbld@aus2-staging.mozilla.org
~/bin/backupsnip 20110124-Firefox-3.6.14-beta #takes 20-50mins - depends if other releases are backing up
=== Push to Beta Channel ===
#From a build slave you can reach aus2-staging
#cltbld@aus2-staging.mozilla.org
#This takes now a lot of time because of PHX datacenter
~/bin/pushsnip 20110124-Firefox-3.6.14-beta
=== Fix snippets for some locales ===
See [https://bugzilla.mozilla.org/show_bug.cgi?id=629256#c2 bug 629256]
cd /opt/aus2/snippets/staging
rsync -a 20110124-Firefox-3.6.14/ 20110126-Firefox-3.6.14-bug629256fix/
touch 20110126-Firefox-3.6.14-bug629256fix/
rm -rf 20110126-Firefox-3.6.14-bug629256fix/Firefox/3.6*/Darwin*/*/{ml,te}
rsync -a 20110124-Firefox-3.5.17-3.6.14-MU/
20110126-Firefox-3.5.17-3.6.14-MU-bug629256fix/
touch 20110126-Firefox-3.5.17-3.6.14-MU-bug629256fix/
rm -rf
20110126-Firefox-3.5.17-3.6.14-MU-bug629256fix/Firefox/3.5*/Darwin*/*/{kn,ml,te}
=== Email mirrors about release day date ===
They need to know 24-48 hours ahead of time.
<pre>
On $DATE1 or $DATE2, we plan to "push to
mirrors" for the Firefox '''$3_6_VERSION''' and Firefox '''$3_5_VERSION''' security releases.
If you believe this will cause any problems, or there are other releases
scheduled for those dates on mirrors, please let us know at
'''$RELEASE_MAILING_LIST'''.
Thanks
'''$NAME'''
</pre>
== Going live day! ==
=== Final checks before push & virus scan ===
Go to the "pre push checks" builder and add these two values:
* '''Revision to build''' FIREFOX_3_6_14_RELEASE
* '''Property 1 name''': release_config: mozilla/release-firefox-mozilla-1.9.2.py
=== Push to Mirrors ===
Skipping; doing a build 2.
=== Final Verification ===
Skipping; doing a build 2.
=== Push release channel snippets ===
Skipping; doing a build 2.
<i>Please use the <tt>20110126-Firefox-3.6.14-bug629256fix</tt> directory when pushing to the release channel for 3.6.x -> 3.6.14 updates, for {{bug|629256}}. We drop updates for a few locales which have strings that make the updater crash. [[User:NThomas|NThomas]] 23:52, 26 January 2011 (PST)</i>
=== Redirect EU Ballot build and add to bouncer ===
Skipping; doing a build 2.
=== Release ===
Skipping; doing a build 2.
=== XULRunner Signing ===
Skipping; doing a build 2.
=== Push XULRunner to mirrors ===
Skipping; doing a build 2.
=== Update XULRunner entries on wiki ===
Skipping; doing a build 2.
=Build 2=
* Clobbered with "Any master/any release/any builder"
* tagged buildbot-configs on '''production''', buildbotcustom on '''production-0.8''', compare-locales and tools with FIREFOX_3_6_14_BUILD2 and FIREFOX_3_6_14_RELEASE
* [ landed build2 changes]
* updated /tools/buildbotcustom/buildbotcustom, /builds/buildbot/configs, and /builds/buildbot/builder_master/tools to latest
* Left tree open, 0.8.0 automation doesn't require that it closes
* kicked off automation with:
PYTHONPATH=/builds/buildbot/builder_master:/builds/buildbot/builder_master/tools/lib/python python /builds/buildbot/builder_master/tools/buildbot-helpers/release_sanity.py -u armenzg -V 3.6.14 --branch mozilla-1.9.2 --build-number 2 -c release-firefox-mozilla-1.9.2.py --dryrun localhost:9010
=== Tag ===
No problems.
I triggered automation twice by mistake so I had to cancel one of the tag jobs.
=== Version bump ===
Not needed as it is build2.


=== Update Bouncer ===
=== Update Bouncer ===
Not needed as it is build2.


=== Stage EU Ballot Builds ===
=== Source / XULRunner Source ===
No problems.
 
=== Build ===
No problems besides:
* win32 was run on an slow IX slave and QA did not get the builds before EOD
 
=== Start autosigning ===
If you are expecting to sign during your workday you can start [https://intranet.mozilla.org/Build:CombinedSigning auto signing] to take care of the deliverables as they are uploaded to FTP.
 
=== L10n Repack ===
No problems.
 
=== Partner Repack ===
No problems.
 
=== XULRunner Build ===
* Linux xulrunner build has a couple of properties going red
** This is not important. The status is red but nothing bad happeneded
** This is a fallout from ctors bug
** Filed as {{bug|632399}}
 
=== Unittests ===
* Found and notified release-drivers of orange found in all 3 platforms
39527 ERROR TEST-UNEXPECTED-FAIL | /tests/content/html/content/test/test_bug481335.xhtml | Should be visited now - "rgb(0, 0, 238)" should equal "rgb(85, 26, 139)"
39528 ERROR TEST-UNEXPECTED-FAIL | /tests/content/html/content/test/test_bug481335.xhtml | Should still be visited after setting pathname to its existing value - "rgb(0, 0, 238)" should equal "rgb(85, 26, 139)"
39531 ERROR TEST-UNEXPECTED-FAIL | /tests/content/html/content/test/test_bug481335.xhtml | Should be visited now after third load - "rgb(0, 0, 238)" should equal "rgb(85, 26, 139)"
 
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.6-Release/1297114970.1297116997.23122.gz
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.6-Release/1297118544.1297120352.5609.gz
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.6-Release/1297131303.1297132922.29438.gz
 
=== Signing ===
Followed [https://intranet.mozilla.org/Build:CombinedSigning CombinedSigning doc].
 
The en-US.xpi change landed on 1.9.1/1.9.2 branches but I forgot that it did not land on the RELBRANCH. Therefore, I forgot again to remove it and I have to recover '''again'''.
 
To recover I did the following:
#ffxbld@stage-old.mozilla.org
rm /home/ftp/pub/mozilla.org/firefox/nightly/3.6.14-candidates/build2/en-US.xpi
 
On the keymaster:
cd ~/signing-work/firefox-3.6.14/signed-build2
rm en-US.xpi MD5SUMS* SHA1SUMS*
cd ..
rm unsigned-build2/en-US.xpi
# http://hg.mozilla.org/build/tools/file/tip/release/signing/Makefile#l73
# NOTE: From the sign2 target we only need to run "checksum-files",
# "create-sigs", "verify-sigs" and "quick-verify"
# http://hg.mozilla.org/build/tools/file/tip/release/signing/Makefile#l218
make checksum-files create-sigs verify-sigs quick-verify stage verify fake-upload PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG} USE_NEW=1
 
* We also hit a signing regression from BUILD1 to BUILD2 - see {{bug|632412}}.
 
To recover I did this:
cd ~/signing-work/firefox-3.6.14
# The next 4 steps are jsut in case we need them for any reason
mkdir -p euballot/signed-build2
mkdir -p euballot/unsigned-build2/unsigned
mv unsigned-build2/unsigned/win32-EUBallot euballot/unsigned-build2/unsigned
mv signed-build2/win32-EUBallot euballot/signed-build2
# Resuming where it bailed out
make verify fake-upload PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG} USE_NEW=1
# Uploading
make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL}
 
So we postponed the signing of the EUBallot until bhearsum run a fix on staging.
These are the steps followed:
cd ~/hg-tools
hg pull
hg up -r FIREFOX_3_6_14_RELEASE
cd ~/signing-work/firefox-3.6.14
cp ~/hg-tools/release/signing/* .
rsync -av -e 'ssh -i /home/cltsign/.ssh/ffxbld_dsa' ffxbld@stage.mozilla.org:/pub/mozilla.org/firefox/nightly/3.6.14-candidates/build2/unsigned/win32-EUBallot/ unsigned-build2/unsigned/win32-EUBallot/
mkdir -p unsigned-build2-euballot-fix/unsigned
rsync -av unsigned-build2/unsigned/win32-EUBallot/ unsigned-build2-euballot-fix/unsigned/win32-EUBallot/
rsync -av --include=unsigned --include=win32 --include=update --include=en-US --include=*.exe --include=*.mar --exclude=* unsigned-build2/unsigned/ unsigned-build2-euballot-fix/unsigned/
./sign-release.py --keydir=${KEYDIR} -j4 --product Firefox -o signed-build2 -p unsigned-build2-euballot-fix
make create-sigs verify fake-upload PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG} USE_NEW=1
# Upon inspection
make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL}
 
Even after the above, EUBallot builds were not properly signed. This time, because of {{bug|632793}}. To fix, did the following:
cd ~/signing-work/firefox-3.6.14
rm -rf cache
# Create a new directory to plop signed ballot builds in, just in case we screw something up
rsync -av --exclude=win32-EUBallot signed-build2/ signed-build2-euballot-attempt2/
# Populate the cache by running sign-release.py w/ --keep-cache and --first-locale against the original directory. This causes it to unpack the first locale's build, cache it, and then exit because there's nothing to sign
for l in `ls unsigned-build2/unsigned/win32-EUBallot`; do ./sign-release.py --keydir=${KEYDIR} -j4 --product Firefox -o signed-build2-euballot-attempt2 -p --first-locale ${l%\/} --keep-cache unsigned-build2/; done
# Now, run sign-release.py on the directory with only the EUBallot builds in it. This will have a 100% hit rate, because all of the files from these locales are in the cache.
./sign-release.py --keydir=${KEYDIR} -j4 --product Firefox -o signed-build2-euballot-attempt2 -p --keep-cache unsigned-build2-euballot-fix/
# Generate detached sigs for them
cd signed-build2-euballot-attempt2
../signing/sign-files .
# Run verify-signatures to ensure everything is OK
./verify-signature.py --product firefox unsigned-build2 signed-build2-euballot-attempt2 > verify-signatures-build2-euballot.fix.log
# Upload!
rsync -e "ssh -i /home/cltsign/.ssh/ffxbld_dsa" -av signed-build2-euballot-attempt2/ ffxbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.6.14-candidates/build2/
 
==== Rename win32-EUBallot for win32-EUballot ====
This is {{bug|633695}} - EUBallot should be renamed to EUballot.
 
There are two patches to fix it but the fix on stage was:
# ffxbld@stage
cd /home/ftp/pub/mozilla.org/firefox/nightly/3.6.14-candidates/build2/
mv win32-EUBallot win32-EUballot
 
=== Notify AV vendors ===
* Notify vendors using instructions in {{bug|568884#c15}}
 
=== L10n Verify ===
I don't even see the point of this builder.
It's always red (expected) and the output does not tell you if there is anything wrong with it.
 
=== Updates ===
 
==== 3.6* -> 3.6.13 ====
===== Generation =====
Failed.
 
Something had landed to fix crashing locales in {{bug|629256}}.
generate-candidate-build-updates.py fails when locales are removed between buildX and buildY of the same release - see {{bug|632449}}
 
We won't be generating buildX to buildY updates and this is how we recovered (from looking at build1 log):
cd /builds/slave/rel-192-updates/
find build/temp/firefox/3.6.13-3.6.14/ftp/firefox/nightly/3.6.14-candidates/build2 -type f -exec chmod 644 {} \;
find build/temp/firefox/3.6.13-3.6.14/ftp/firefox/nightly/3.6.14-candidates/build2 -type d -exec chmod 755 {} \;
cd /builds/slave/rel-192-updates/build/temp/firefox/3.6.13-3.6.14/ftp/firefox/nightly/3.6.14-candidates/build2
rsync -av -e 'ssh -oIdentityFile=~/.ssh/ffxbld_dsa' --exclude=*complete.mar update ffxbld@stage-old.mozilla.org:/pub/mozilla.org/firefox/nightly/3.6.14-candidates/build2/
cd /builds/slave/rel-192-updates/build/temp/firefox/3.6.13-3.6.14
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20110208-Firefox-3.6.14-test
cd /builds/slave/rel-192-updates/build
bash -c 'ssh -l cltbld -i ~/.ssh/cltbld_dsa aus2-staging.mozilla.org ~/bin/backupsnip 20110208-Firefox-3.6.14-test'
bash -c 'ssh -l cltbld -i ~/.ssh/cltbld_dsa aus2-staging.mozilla.org ~/bin/pushsnip 20110208-Firefox-3.6.14-test'
cd /builds/slave/rel-192-updates/build/temp/firefox/3.6.13-3.6.14
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20110208-Firefox-3.6.14
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20110208-Firefox-3.6.14-beta
bash /builds/slave/rel-192-updates/tools/release/compare-channel-snippets.sh aus2 release aus2.test releasetest
 
===== Update Verify =====
"Force build" was used since the updates builder failed.
 
* linux and windows update verify succeded
* mac update verify failed for "ml" and "te"
 
I grep the log with "grep -E (FAIL|Using)" and extracted the blocks that had failures:
<pre>
Using  https://aus2.mozilla.org/update/1/Firefox/3.6.13/20101203074432/Darwin_Universal-gcc3/ml/betatest/update.xml?force=1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Darwin_Universal-gcc3 downloads/Firefox 3.6.13.dmg vs. downloads/Firefox 3.6.14.dmg: 1
Using  https://aus2.mozilla.org/update/1/Firefox/3.6.13/20101203074432/Darwin_Universal-gcc3/ml/betatest/update.xml?force=1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Darwin_Universal-gcc3 downloads/Firefox 3.6.13.dmg vs. downloads/Firefox 3.6.14.dmg: 1
 
Using  https://aus2.mozilla.org/update/1/Firefox/3.6.13/20101203074432/Darwin_Universal-gcc3/te/betatest/update.xml?force=1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Darwin_Universal-gcc3 downloads/Firefox 3.6.13.dmg vs. downloads/Firefox 3.6.14.dmg: 1
Using  https://aus2.mozilla.org/update/1/Firefox/3.6.13/20101203074432/Darwin_Universal-gcc3/te/betatest/update.xml?force=1
FAIL: binary files found in diff
FAIL: check_updates returned failure for Darwin_Universal-gcc3 downloads/Firefox 3.6.13.dmg vs. downloads/Firefox 3.6.14.dmg: 1
</pre>
 
=== Socorro product data updates ===
Already done for '''build 1'''
 
== Going to beta day!! ==
Do this in the morning before the rel-driver gives you the GO.
#From a build slave you can reach aus2-staging
#cltbld@aus2-staging.mozilla.org
cd /opt/aus2/snippets/staging/
~/bin/backupsnip 20110208-Firefox-3.6.14-beta # takes more than 45 minutes


=== Push to Beta Channel ===
=== Push to Beta Channel ===
#From a build slave you can reach aus2-staging
#cltbld@aus2-staging.mozilla.org
#This takes now a lot of time because of PHX datacenter
cd /opt/aus2/snippets/staging/
~/bin/pushsnip 20110208-Firefox-3.6.14-beta
=== Fix snippets for some locales ===
Nothing to be done since nthomas landed some fixes in [https://bugzilla.mozilla.org/show_bug.cgi?id=629256#c10 bug 629256]
=== Email mirrors about release day date ===
They need to know 24-48 hours ahead of time.
<pre>
On $DATE1 or $DATE2, we plan to "push to
mirrors" for the Firefox '''$3_6_VERSION''' and Firefox '''$3_5_VERSION''' security releases.
If you believe this will cause any problems, or there are other releases
scheduled for those dates on mirrors, please let us know at
'''$RELEASE_MAILING_LIST'''.
Thanks
'''$NAME'''
</pre>
== Going live day! ==
=== Final checks before push & virus scan ===
Go to the "pre push checks" builder and add these two values:
* '''Revision to build''' FIREFOX_3_6_14_RELEASE
* '''Property 1 name''': release_config: mozilla/release-firefox-mozilla-1.9.2.py
* It failed the first time because of permission issues. {{bug|633961}}
* I triggered it a second time
Skipping; doing a build 3.
=== Push to Mirrors ===
Skipping; doing a build 3.
=== Final Verification ===
Skipping; doing a build 3.
=== Push release channel snippets ===
Skipping; doing a build 3.
=== Push EU Ballot build ===
Skipping; doing a build 3.
=== Release ===
Skipping; doing a build 3.
=== XULRunner Signing ===
Skipping; doing a build 3.
=== Push XULRunner to mirrors ===
Skipping; doing a build 3.
=== Update XULRunner entries on wiki ===
Skipping; doing a build 3.
=Build 3=
'''NOTE: We might not send this build to the users but build#2. This will be decided on Tuesday'''
* Clobbered with "Any master/any release/any builder"
* tagged with FIREFOX_3_6_14_BUILD3 and FIREFOX_3_6_14_RELEASE
** buildbot-configs on '''production'''
** buildbotcustom on '''production-0.8'''
** compare-locales & tools
* [http://hg.mozilla.org/build/buildbot-configs/rev/517992f6aa54 landed build2 changes]
* updated /tools/buildbotcustom/buildbotcustom, /builds/buildbot/configs, and /builds/buildbot/builder_master/tools to latest
* Left tree open, 0.8.0 automation doesn't require that it closes
* kicked off automation with:
PYTHONPATH=/builds/buildbot/builder_master:/builds/buildbot/builder_master/tools/lib/python python /builds/buildbot/builder_master/tools/buildbot-helpers/release_sanity.py -u armenzg -V 3.6.14 --branch mozilla-1.9.2 --build-number 3 -c release-firefox-mozilla-1.9.2.py --dryrun localhost:9010
=== Tag ===
No problems.
=== Version bump ===
Not needed as it is build3.
=== Update Bouncer ===
Not needed as it is build3.
=== Source / XULRunner Source ===
No problems.
=== XULRunner Build ===
No problems.
=== Build ===
No problems.
=== Start autosigning ===
If you are expecting to sign during your workday you can start [https://intranet.mozilla.org/Build:CombinedSigning auto signing] to take care of the deliverables as they are uploaded to FTP.
=== L10n Repack ===
No problems.
* repack 5/6 failed due to checkout timeout on slow slave
** re-triggered
=== Partner Repack ===
No problems.


=== Unittests ===
All three platforms fails with these:
39509 ERROR TEST-UNEXPECTED-FAIL | /tests/content/html/content/test/test_bug481335.xhtml | Should be visited now - "rgb(0, 0, 238)" should equal "rgb(85, 26, 139)"
39510 ERROR TEST-UNEXPECTED-FAIL | /tests/content/html/content/test/test_bug481335.xhtml | Should still be visited after setting pathname to its existing value - "rgb(0, 0, 238)" should equal "rgb(85, 26, 139)"


Notified LegNeato.


=== 3.0.19 -> 3.6.13 Major Update ===
=== Signing ===
==== Generation ====
Followed [https://intranet.mozilla.org/Build:CombinedSigning CombinedSigning doc].


==== Quick Verification ====
=== Notify AV vendors ===
* Notify vendors using instructions in {{bug|568884#c15}}


==== Push to AUS ====
=== L10n Verify ===
I don't even see the point of this builder.
It's always red (expected) and the output does not tell you if there is anything wrong with it.


==== Push test snippets ====
=== Updates ===


==== Verify ====
==== 3.6* -> 3.6.13 ====
===== Generation =====
We landed a fix to avoid this from happening but I forgot to reconfigure.


==== Results ====
cd /builds/slave/rel-192-updates/
find build/temp/firefox/3.6.13-3.6.14/ftp/firefox/nightly/3.6.14-candidates/build3 -type f -exec chmod 644 {} \;
find build/temp/firefox/3.6.13-3.6.14/ftp/firefox/nightly/3.6.14-candidates/build3 -type d -exec chmod 755 {} \;
cd /builds/slave/rel-192-updates/build/temp/firefox/3.6.13-3.6.14/ftp/firefox/nightly/3.6.14-candidates/build3
rsync -av -e 'ssh -oIdentityFile=~/.ssh/ffxbld_dsa' --exclude=*complete.mar update ffxbld@stage-old.mozilla.org:/pub/mozilla.org/firefox/nightly/3.6.14-candidates/build3/
cd /builds/slave/rel-192-updates/build/temp/firefox/3.6.13-3.6.14
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20110219-Firefox-3.6.14-test
cd /builds/slave/rel-192-updates/build
bash -c 'ssh -l cltbld -i ~/.ssh/cltbld_dsa aus2-staging.mozilla.org ~/bin/backupsnip 20110219-Firefox-3.6.14-test'
bash -c 'ssh -l cltbld -i ~/.ssh/cltbld_dsa aus2-staging.mozilla.org ~/bin/pushsnip 20110219-Firefox-3.6.14-test'
cd /builds/slave/rel-192-updates/build/temp/firefox/3.6.13-3.6.14
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20110219-Firefox-3.6.14
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/2011019-Firefox-3.6.14-beta
bash /builds/slave/rel-192-updates/tools/release/compare-channel-snippets.sh aus2 release aus2.test releasetest


=== Stage EU Ballot Builds ===
===== Update Verify =====
Same as with update verify for build2. 'mk' and 'te' failed for mac.
 
See {{bug|629256}} for more details.
 
=== Socorro product data updates ===
Already done for '''build 1'''
 
== Going to beta day!! ==
Do this in the morning before the rel-driver gives you the GO.
#From a build slave you can reach aus2-staging
#cltbld@aus2-staging.mozilla.org
cd /opt/aus2/snippets/staging/
~/bin/backupsnip 20110219-Firefox-3.6.14-beta # takes more than 45mins
 
To make backupsnip to work we had to re-enable updates ({{bug|636218}}):
#on aus2-staging and phx
cd /opt/aus2/incoming/3/Firefox/
chmod 700 -v 3.6*
 
=== Push to Beta Channel ===
#From a build slave you can reach aus2-staging
#cltbld@aus2-staging.mozilla.org
#This takes now a lot of time because of PHX datacenter
cd /opt/aus2/snippets/staging/
~/bin/pushsnip 20110208-Firefox-3.6.14-beta
 
=== Email mirrors about release day date ===
They need to know 24-48 hours ahead of time.
 
<pre>
On $DATE1 or $DATE2, we plan to "push to
mirrors" for the Firefox '''$3_6_VERSION''' and Firefox '''$3_5_VERSION''' security releases.
 
If you believe this will cause any problems, or there are other releases
scheduled for those dates on mirrors, please let us know at
'''$RELEASE_MAILING_LIST'''.
 
Thanks
'''$NAME'''
</pre>
 
== Going live day! ==
 
=== Final checks before push & virus scan ===
I trigger the builders last week and it all went well.
 
I trigger it once more manually to re-verify it.
clamdscan /mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/3.6.14-candidates/build3/
/mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/3.6.14-candidates/build3: OK
----------- SCAN SUMMARY -----------
Infected files: 0
Time: 5466.948 sec (91 m 6 s)


=== Virus Scan ===
=== Backup release snippets ===
Do this in the morning ahead of time before the "push snippets to release".
#From a build slave you can reach aus2-staging
#cltbld@aus2-staging.mozilla.org
cd /opt/aus2/snippets/staging/
~/bin/backupsnip 20110219-Firefox-3.6.14


=== Final checks before push ===
=== Push to Mirrors and modify exclude file ===
After some re-tagging and reconfigures for getting some changes that had landed on default we manage to trigger this builder and succeed to completion.
http://hg.mozilla.org/build/buildbotcustom/rev/03084b14ff4b


=== Push to Mirrors ===
Push to community mirrors:
# cltbld@stage
vim -n /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude
# replace 3.6.13 with 3.6.14


=== Final Verification ===
=== Final Verification ===
Triggered automatically.


=== Push release channel snippets ===
=== Push release channel snippets ===
The backupsnip was done earlier in the [[Releases/Firefox_3.6.14/BuildNotes#Backup_release_snippets backup release snippets]] section.
#From a build slave you can reach aus2-staging
#cltbld@aus2-staging.mozilla.org
cd /opt/aus2/snippets/staging/
~/bin/pushsnip 20110219-Firefox-3.6.14
=== Redirect EU Ballot build and add bouncer entry ===
Reopened {{bug|632052}} for IT to fix it when the snippets are live


=== Push 3.0.19 -> 3.6.13 MU snippets ===
NOTE: I think there is a bug that got fixed recently that adds the bouncer entry for the EUballot builds automatically. I can't find it right now.


=== Push EU Ballot build ===
* Filed {{bug|632052}} to change download redirect
* On bouncer we add "Firefox-3.6.14-EUballot" product under "Mirrors->Products->Add new product"
* Once added, we add a new mirror location:
/firefox/releases/3.6.14/win32-EUballot/:lang/Firefox%20Setup%203.6.14.exe


=== Release ===
=== Release ===
Update symlinks
#ffxbld@stage
cd /home/ftp/pub/firefox/releases
rm latest-3.6 && ln -s 3.6.14 latest-3.6
Sent the following mail to metrics@mozilla.com:
Firefox,firefox,firefox,3.6.13,3.6
Firefox,firefox,firefox,3.6.14pre,3.6


=== XULRunner Signing ===
=== XULRunner Signing ===
Create ~/signXR1.9.2.14.bash file with chmod +x and this content:
<pre>
#!/bin/bash
set -x
set -e
PRODUCT=xulrunner
VERSION=1.9.2.14
BUILD=3
TAG=FIREFOX_3_6_14_RELEASE
REPO="releases/mozilla-1.9.2"
EMAIL=armenzg@mozilla.com
KEYDIR=d:/2010-keys
cd ~/hg-tools
hg pull
hg up -r ${TAG}
mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
make setup PRODUCT=${PRODUCT} VERSION=${VERSION} \
  BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  --exclude=*.txt --exclude=*-symbols.zip \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/ \
unsigned-build${BUILD}
rsync -av --exclude=unsigned unsigned-build${BUILD}/ \
signed-build${BUILD}/
rsync -av unsigned-build${BUILD}/unsigned/ \
signed-build${BUILD}/
make checksum-files create-sigs stage verify-sigs \
  PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} \
  REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
rmdir signed-build${BUILD}/contrib{,-localized}
rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  signed-build${BUILD}/ \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/
</pre>


=== Push XULRunner to mirrors ===
=== Push XULRunner to mirrors ===
# xrbld@stage
rsync -av --exclude=*.txt --exclude=*unsigned*  --exclude=*-symbols.zip \
  /pub/mozilla.org/xulrunner/nightly/1.9.2.14-candidates/build3/ \
  /pub/mozilla.org/xulrunner/releases/1.9.2.14/
Edited cltbld@stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude and added
- xulrunner/releases/1.9.2.13


=== Update XULRunner entries on wiki ===
=== Update XULRunner entries on wiki ===
Updated the links in these two wiki pages for the version change:
* [https://developer.mozilla.org/En/XULRunner_1.9.2_Release_Notes Release Notes]
* [https://developer.mozilla.org/En/Gecko_SDK Gecko_SDK]
Armenzg
Confirmed users
3,990

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/279617...288022"