Releases/Firefox 18.0/BuildNotes
Build 1
Checklist
Ship the l10n milestone DESKTOP MOBILE RELEASE BETA- Land patches and reconfigDESKTOP MOBILE RELEASE BETA ESR
- Start the automation DESKTOP MOBILE RELEASE BETA ESR
- Sign Android builds MOBILE RELEASE BETA
- Push to mirrors DESKTOP RELEASE
- Publish Fennec MOBILE RELEASE BETA NB: Push armv6 apk as well, like we do for betas
- Run pushsnip DESKTOP RELEASE BETA ESR
Update bouncer links DESKTOP BETA RELEASE- Update XULRunner wiki pages DESKTOP RELEASE
Post-release tasks DESKTOP RELEASE BETA ESRGet the Browser Choice website updated DESKTOP RELEASE
Notes
Tracking bug 810325 release engineer kmoir
Release was submitted using release-runner
bugs hit
bug 826825 Remove mn, sw during beta-release migration for Firefox 18
A bug wasn't opened to remove the mn and sw locales for the release, so I removed them manually in the changesets submitted to the release runner. However, they still existed in https://hg.mozilla.org/releases/mozilla-release/file/3a7b66445659/browser/locales/shipped-locales so relman had to fix this and issue a new changeset to us. Once this was releasesd, bhearsum retagged our repos and I reran the tagging builds that failed.
kmoir, jhopkins: so, we all need to "force build" our "source", "build", "bouncer submitter", and "xulrunner build" builders (if you have them) because buildbot schedulers are dumb
source
build win32 macosx64 linux linux64
bouncer submitter
xulrunner win32 macosx64 linux linux64
Fennec
source
build android android-armv6
xulrunner_macosx64 had a compile error, bhearsum identfied it as a known intermittent issue and reran the build
should check the Fennec build on Monday and ensure the build ids are okay for shipping.
Because of rebuilding tag instead of re-doing sendchanges, all repacks are failing because "script_repo_revision" isn't set. These can be fixed by forcing the failed builders and setting a "script_repo_revision" property to the release tag (eg, FIREFOX_18_0_RELEASE). I've done this for all failed repacks that I've seen. (Though, some have additional problems, see below for details).
Linux repacks failed with "cannot create executable" during configure. This is because the gcc path is incorrect. The mozconfigs were updated for official betas recently for mock slaves, but we missed the esr's -release dir's:
http://hg.mozilla.org/build/buildbot-configs/rev/4f68d76200ef#l16.1
Catlee fixed this http://diff.pastebin.mozilla.org/2039996.
The android repacks were failing because bug 814191 wasn't landed. I didn't realize that this had to be landed before the beta. I landed this patch, catlee landed the buildbot configs and retagged them and restarted the failed repacks.
catlee and I landed two patches: http://hg.mozilla.org/build/buildbot-configs/rev/e8b7870b1d96 http://hg.mozilla.org/build/mozharness/rev/522935ecb3c1
catlee re-tagged buildbot-configs with:
FENNEC_18_0_RELEASE FENNEC_18_0_BUILD1 FIREFOX_18_0_RELEASE FIREFOX_18_0_BUILD1
catlee re-triggered:
Firefox 18 32/64-bit linux repacks Fennec 18 repacks
When signing the Android builds, this message appeared
5:18:53 ERROR - Unable to download android:multi unsigned apk! 15:18:53 ERROR - Unable to download android-armv6:multi unsigned apk! 15:18:53 ERROR - Downloaded 29 of 31 unsigned apks successfully. 15:18:53 INFO - Signed 29 of 29 apks successfully. 15:18:53 INFO - Copying logs to upload dir... 15:18:53 INFO - mkdir: /home/cltsign/signing-work/release/upload/logs
Don't recall that from before, not sure if this is normal.
No, it's not, nthomas opened bug 827224
Fennec multilocale
This was caused by a missing --package-multi option in the worst possible place.
Happily, mozharness allows you to run actions separately, and we don't clobber release directories by default.
- Looked at the logs; we used bld-centos6-hp-007 and 016 for android-armv6 and android, respectively.
- Disabled in slavealloc temporarily
- Backed up objdir
# armv6 cltbld@bld-centos6-hp-007 cd /builds/slave/rel-m-rel-andrd-armv6-bld/build tar czvf obj-firefox.tar.gz obj-firefox
# armv7 cltbld@bld-centos6-hp-016 cd /builds/slave/rel-m-rel-andrd-bld/build tar czvf obj-firefox.tar.gz obj-firefox
- Tried to package-multi (from log, but with actions changed to --package-multi --summary)
mock_mozilla -v -r mozilla-centos6-i386 --cwd /builds/slave/rel-m-rel-andrd-armv6-bld --unpriv --shell '/usr/bin/env MOZ_OBJDIR="obj-firefox" PATH="/tools/buildbot/bin:/usr/local/bin:/bin:/usr/bin" SYMBOL_SERVER_USER="ffxbld" DISPLAY=":2" CCACHE_UMASK="002" SHIP_LICENSED_FONTS="1" MOZ_SYMBOLS_EXTRA_BUILDID="android-armv6" POST_SYMBOL_UPLOAD_CMD="/usr/local/bin/post-symbol-upload.py" SYMBOL_SERVER_PATH="/mnt/netapp/breakpad/symbols_mob/" MOZILLA_REV="FENNEC_18_0_RELEASE" JAVA_HOME="/tools/jdk6" HG_SHARE_BASE_DIR="/builds/hg-shared" LC_ALL="C" SYMBOL_SERVER_HOST="symbols1.dmz.phx1.mozilla.com" TINDERBOX_OUTPUT="1" COMM_REV="FENNEC_18_0_RELEASE" CCACHE_DIR="/builds/ccache" CCACHE_COMPRESS="1" MOZ_UPDATE_CHANNEL="release" SYMBOL_SERVER_SSH_KEY="/home/mock_mozilla/.ssh/ffxbld_dsa" JARSIGNER="/builds/slave/rel-m-rel-andrd-armv6-bld/tools/release/signing/mozpass.py" MOZ_CRASHREPORTER_NO_REPORT="1" python mozharness/scripts/multil10n.py --config-file multi_locale/release_mozilla-release_android-armv6.json --merge-locales --tag-override=FENNEC_18_0_RELEASE --package-multi --summary'
This failed out because of a missing 'aapt' in the mock env. Poked around a bit, figured I needed to recreate the mock environment for Android.
- Recreated mock env (copy-pasted from previous log)
# cltbld@bld-centos6-hp-007 and 016 sh -c 'rm -f /builds/mock_mozilla/mozilla-centos6-i386/buildroot.lock; mock_mozilla -v -r mozilla-centos6-i386 --orphanskill' mock_mozilla -v -r mozilla-centos6-i386 --init mock_mozilla -r mozilla-centos6-i386 --copyin /home/cltbld/.ssh /home/mock_mozilla/.ssh mock_mozilla -v -r mozilla-centos6-i386 --cwd / --shell '/usr/bin/env chown -R mock_mozilla /home/mock_mozilla/.ssh' mock_mozilla -r mozilla-centos6-i386 --copyin /home/cltbld/.android /builds/.android mock_mozilla -v -r mozilla-centos6-i386 --cwd / --shell '/usr/bin/env chown -R mock_mozilla /builds/.android' mock_mozilla -r mozilla-centos6-i386 --copyin /home/cltbld/.mozpass.cfg /builds/.mozpass.cfg mock_mozilla -v -r mozilla-centos6-i386 --cwd / --shell '/usr/bin/env chown -R mock_mozilla /builds/.mozpass.cfg' mock_mozilla -r mozilla-centos6-i386 --copyin /home/cltbld/.hgrc /builds/.hgrc mock_mozilla -v -r mozilla-centos6-i386 --cwd / --shell '/usr/bin/env chown -R mock_mozilla /builds/.hgrc' mock_mozilla -v -r mozilla-centos6-i386 --cwd / --unpriv --shell '/usr/bin/env mkdir -p /builds/slave/rel-m-rel-andrd-bld/build' mock_mozilla -v -r mozilla-centos6-i386 --install autoconf213 mozilla-python27-mercurial ccache android-sdk15 android-sdk16 android-ndk5 android-ndk8 zip java-1.6.0-openjdk-devel zlib-devel glibc-static openssh-clients mpfr wget
- Packaged multi
# android-armv6 cltbld@bld-centos6-hp-007 mock_mozilla -v -r mozilla-centos6-i386 --cwd /builds/slave/rel-m-rel-andrd-armv6-bld --unpriv --shell '/usr/bin/env MOZ_OBJDIR="obj-firefox" PATH="/tools/buildbot/bin:/usr/local/bin:/bin:/usr/bin" SYMBOL_SERVER_USER="ffxbld" DISPLAY=":2" CCACHE_UMASK="002" SHIP_LICENSED_FONTS="1" MOZ_SYMBOLS_EXTRA_BUILDID="android-armv6" POST_SYMBOL_UPLOAD_CMD="/usr/local/bin/post-symbol-upload.py" SYMBOL_SERVER_PATH="/mnt/netapp/breakpad/symbols_mob/" MOZILLA_REV="FENNEC_18_0_RELEASE" JAVA_HOME="/tools/jdk6" HG_SHARE_BASE_DIR="/builds/hg-shared" LC_ALL="C" SYMBOL_SERVER_HOST="symbols1.dmz.phx1.mozilla.com" TINDERBOX_OUTPUT="1" COMM_REV="FENNEC_18_0_RELEASE" CCACHE_DIR="/builds/ccache" CCACHE_COMPRESS="1" MOZ_UPDATE_CHANNEL="release" SYMBOL_SERVER_SSH_KEY="/home/mock_mozilla/.ssh/ffxbld_dsa" JARSIGNER="/builds/slave/rel-m-rel-andrd-armv6-bld/tools/release/signing/mozpass.py" MOZ_CRASHREPORTER_NO_REPORT="1" python mozharness/scripts/multil10n.py --config-file multi_locale/release_mozilla-release_android-armv6.json --merge-locales --tag-override=FENNEC_18_0_RELEASE --package-multi --summary'
# android cltbld@bld-centos6-hp-016 mock_mozilla -v -r mozilla-centos6-i386 --cwd /builds/slave/rel-m-rel-andrd-bld --unpriv --shell '/usr/bin/env HG_SHARE_BASE_DIR="/builds/hg-shared" JARSIGNER="/builds/slave/rel-m-rel-andrd-bld/tools/release/signing/mozpass.py" SHIP_LICENSED_FONTS="1" MOZ_CRASHREPORTER_NO_REPORT="1" SYMBOL_SERVER_HOST="symbols1.dmz.phx1.mozilla.com" CCACHE_DIR="/builds/ccache" POST_SYMBOL_UPLOAD_CMD="/usr/local/bin/post-symbol-upload.py" MOZILLA_REV="FENNEC_18_0_RELEASE" SYMBOL_SERVER_SSH_KEY="/home/mock_mozilla/.ssh/ffxbld_dsa" PATH="/tools/jdk6/bin:/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/" MOZ_UPDATE_CHANNEL="release" TINDERBOX_OUTPUT="1" CCACHE_COMPRESS="1" COMM_REV="FENNEC_18_0_RELEASE" MOZ_OBJDIR="obj-firefox" SYMBOL_SERVER_PATH="/mnt/netapp/breakpad/symbols_mob/" LC_ALL="C" SYMBOL_SERVER_USER="ffxbld" JAVA_HOME="/tools/jdk6" DISPLAY=":2" CCACHE_UMASK="002" python mozharness/scripts/multil10n.py --config-file multi_locale/release_mozilla-release_android.json --merge-locales --tag-override=FENNEC_18_0_RELEASE --package-multi --summary'
- Uploaded multi
# android-armv6 cltbld@bld-centos6-hp-007 mock_mozilla -v -r mozilla-centos6-i386 --cwd /builds/slave/rel-m-rel-andrd-armv6-bld/build/obj-firefox --unpriv --shell '/usr/bin/env MOZ_OBJDIR="obj-firefox" PATH="/tools/buildbot/bin:/usr/local/bin:/bin:/usr/bin" SYMBOL_SERVER_USER="ffxbld" DISPLAY=":2" CCACHE_UMASK="002" SHIP_LICENSED_FONTS="1" MOZ_SYMBOLS_EXTRA_BUILDID="android-armv6" POST_SYMBOL_UPLOAD_CMD="/usr/local/bin/post-symbol-upload.py" SYMBOL_SERVER_PATH="/mnt/netapp/breakpad/symbols_mob/" UPLOAD_USER="ffxbld" MOZILLA_REV="FENNEC_18_0_RELEASE" JAVA_HOME="/tools/jdk6" HG_SHARE_BASE_DIR="/builds/hg-shared" SYMBOL_SERVER_HOST="symbols1.dmz.phx1.mozilla.com" UPLOAD_SSH_KEY="~/.ssh/ffxbld_dsa" TINDERBOX_OUTPUT="1" COMM_REV="FENNEC_18_0_RELEASE" MOZ_CRASHREPORTER_NO_REPORT="1" LC_ALL="C" UPLOAD_TO_TEMP="1" MOZ_UPDATE_CHANNEL="release" SYMBOL_SERVER_SSH_KEY="/home/mock_mozilla/.ssh/ffxbld_dsa" UPLOAD_EXTRA_FILES="android-armv6_info.txt android-armv6_info.txt" UPLOAD_HOST="stage.mozilla.org" POST_UPLOAD_CMD="post_upload.py -p mobile -n 1 -v 18.0 --builddir android-armv6/multi --release-to-mobile-candidates-dir --nightly-dir=candidates" CCACHE_DIR="/builds/ccache" CCACHE_COMPRESS="1" python '"'"'/builds/slave/rel-m-rel-andrd-armv6-bld/tools/buildfarm/utils/retry.py'"'"' -s 1 -r 5 -t 3660 make upload AB_CD=multi'
# android cltbld@bld-centos6-hp-016 mock_mozilla -v -r mozilla-centos6-i386 --cwd /builds/slave/rel-m-rel-andrd-bld/build/obj-firefox --unpriv --shell '/usr/bin/env MOZ_OBJDIR="obj-firefox" PATH="/tools/jdk6/bin:/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/" SYMBOL_SERVER_USER="ffxbld" DISPLAY=":2" CCACHE_UMASK="002" SHIP_LICENSED_FONTS="1" UPLOAD_USER="ffxbld" POST_SYMBOL_UPLOAD_CMD="/usr/local/bin/post-symbol-upload.py" SYMBOL_SERVER_PATH="/mnt/netapp/breakpad/symbols_mob/" MOZILLA_REV="FENNEC_18_0_RELEASE" JAVA_HOME="/tools/jdk6" HG_SHARE_BASE_DIR="/builds/hg-shared" SYMBOL_SERVER_HOST="symbols1.dmz.phx1.mozilla.com" UPLOAD_SSH_KEY="~/.ssh/ffxbld_dsa" TINDERBOX_OUTPUT="1" COMM_REV="FENNEC_18_0_RELEASE" MOZ_CRASHREPORTER_NO_REPORT="1" LC_ALL="C" UPLOAD_TO_TEMP="1" MOZ_UPDATE_CHANNEL="release" SYMBOL_SERVER_SSH_KEY="/home/mock_mozilla/.ssh/ffxbld_dsa" UPLOAD_EXTRA_FILES="android_info.txt android_info.txt" UPLOAD_HOST="stage.mozilla.org" POST_UPLOAD_CMD="post_upload.py -p mobile -n 1 -v 18.0 --builddir android/multi --release-to-mobile-candidates-dir --nightly-dir=candidates" CCACHE_DIR="/builds/ccache" CCACHE_COMPRESS="1" python '"'"'/builds/slave/rel-m-rel-andrd-bld/tools/buildfarm/utils/retry.py'"'"' -s 1 -r 5 -t 3660 make upload AB_CD=multi'
- Re-signed multi. This skipped all single locales, but re-signed en-US as well as multi.
# cltsign@signing1 cd signing-work python2.6 mozharness/scripts/sign_android.py --config-file signing/android_mozilla-release.py --locale multi
- Reenabled slaves in slavealloc
- Rebooted slaves
- Pinged ioanachiorean in #build and sent r-d email.
Run euballot bouncer submitter
QA found that the euballot bouncer links weren't working. After some investigation we discovered that this was because the euballot bouncer entries were never added. That builder never ran because of the problems with tagging, and we forgot to trigger it. Used "force build" to start the builder and fix the problem.