View source for Release:Release Automation on Mercurial:Updates through Shipping

{{Release:DocMainHeader}}

= Update/snippet Generation =
Deliverables:
* Updated patcher config file
* Updated update verify configs
* Release blobs submitted to production Balrog
* Updated test channel rules in Balrog, pointing at the new release Blob

After checking out the necessary modules and tools this builder goes to work doing the following:
* Bumping & checking in a patcher config file
* Creating & uploading update snippets

= Set-up whatsnew page =
At the time of writing, manual work is required if we're shipping a release that has a mix of old versions that need/don't need a whatsnew page. If Release Management hasn't specified that a whatsnew page is needed, there's nothing to do. This must be done after initial update submission is done, and you'll need to know which where the cut off line for the whatsnew is. For example, when 35.0 shipped everything earlier than 34.0 required a whatsnew page, but 34.0 and 34.0.5 did not. If you don't grok how Balrog rules work it would be a good idea [[Balrog#Rules | to read up on them]] before getting started here.

Note: do not modify blobs until both "updates" builders have finished. Your changes may be overwritten otherwise.

To create a blob with the whatsnew page enabled, do the following:
* Load the [https://aus4-admin.mozilla.org/ui/ Balrog Admin UI].
* Go to the "Releases" section.
* Find the release you're working on by filtering.
* Click "Download".
* Edit the local file and add the following at the top level of the blob (replace the VERSION variable with the version of the release, do NOT replace %LOCALE% or %OLD_VERSION% - those are substituted by Balrog/Firefox):
 "openURL": "https://www.mozilla.org/%LOCALE%/firefox/$MANUALLY_REPLACE_THIS_VARIABLE_CALLED_VERSION/whatsnew/?oldversion=%OLD_VERSION%",
 "actions": "showURL",
* Change the "name" in the blob to something unique (usually by appending "-whatsnew" to it).
* Back in the Admin UI, click "Add a new release".
* The name should match the one you just edited in the blob, product should be Firefox.
* Upload the new blob.

Now that you've got the the new blob you need to adjust rules to point at it. This is usually done in two stage: once for the test channels, and once for the live channel. Test channel updates are typically done as soon as the automated channel updates become available. Live channel updates typically happen at the same time we push the release live. Take care when modifying rules - any changes you make to live channels take effect immediately.

Because the changes that need to be made are highly dependent on the state of the rules at the time and the specific versions requiring/not requiring a whatsnew page, specific step-by-step instructions are explicitly not provided. You may need to manipulate existing rules, add new ones, or even delete some. As an example, here's what the release live and test channels rules looked like after 35.0 was shipped:<br/>
[[File:Balrog_whatsnew_rules_example.png]]

The rules above are listed from highest priority to lowest. The first is ignorable for our purposes (it's a long standing watershed update rule for users on versions earlier than 10.0). The next 3 are are rules that point to the whatsnew version of the blob created above - one rule each for the cdntest, localtest, and live channels. The last 3 point users at the non-whatsnew blob that was fully created by automation. Note that the whatsnew rules specify a version ("<34.0") and have a higher priority.

= Update Verify =
Code used:
* [http://hg.mozilla.org/build/tools/file/tip/release/updates/verify.sh release/updates/verify.sh]

Deliverables:
* Update verify logs per platform. Details on them are below.

Update verify, in its current form, verifies a few different things:
* That the "updater" binary works
* That all MAR files can be applied, and result in a functionally equivalent build to the installer.
* That all previous versions (across all locales), have snippets available

It does this in two different types of tests:
* Full update verify unpacks a previous version's installer, pings AUS, downloads the MAR it finds, applies it, and diffs the result against the current version's installer. Differences between the two are reported in the log.
** Run for all locales in the oldVersion
** Run for select locales ("de", "en-US", "ru" at the time of writing) for all other previous versions
** Completes and partials (if present) are both tested
* Quick update verify does a subset of the above: it pings AUS, downloads the MAR it finds, and ensures that it is non-zero in size
** Run against previous versions for all locales not tested as part of the full update verify.

Update verify is split into multiple chunks per platform (10, by default). The full and quick checks are spaced evenly throughout the chunks giving each one roughly equivalent run time.

== Log Analysis ==
This step is pretty good at reporting errors. Most errors will turn the step red, and be pretty clearly visible in the log. Nonetheless it's a good idea to have a scan through the log for anything abnormal (searching for FAIL and WARN are helpful here).

Here's some common failures:
* "Binary files source/bin/freebl3.chk and target/bin/freebl3.chk differ" (or softokn3.chk) on win32 in the complete MAR
** This is normal in the win32 logs and is caused by {{bug|404340}}. Once this bug is fixed it should go away.
* "CRC check failed" and/or "LoadSourceFile failed" on a partial MAR
** This typically happens when the previous release's complete MAR differs from the complete package. Because partial MARs are generated by diffing the previous complete MAR with the current complete MAR the partial will fail the CRC check when this happens. There's nothing that can be done about this, because the problem is with an already-release-build. The correct thing to do here is inform release-drivers that there will not be partials available on this platform, trim them out of the candidates directory, and remove the partial snippets.
* "FAIL: partial from xxxxx wrong size", "FAIL: actual size: 0"
** This is usually caused by missing MARs. You should first check to see if the MAR in question exists in the candidates directory. If it doesn't, go back and look at the logs to see if there was a problem generating or uploading it. If it does exist in the candidates directory make sure the permissions are set properly (ffxbld:firefox, 664).
* "FAIL: Could not download source xxx"
** Usually indicates missing packages or bad permissions in the candidates directory. Take the same action as above.
* "FAIL: no partial update found for xxx" or "FAIL: no complete update found for xxx"
** Check the logs from the previous step to make sure 'pushsnip' was run successfully. This error usually indicates that snippets weren't pushed.

= Check Permissions =
**only applicable for non release promotion releases**
This builder:
# Checks if the FTP directories and files have proper permissions and modes
# Runs rsync in dry-run mode

This builder is triggered automatically after the updates builder.

No reconfig needed for this builder in manual mode. Automatic mode requires reconfig only on version bump.

= Antivirus check =
**only applicable for non release promotion releases. release promotion bundles antivirus within beetmover tasks**
This builder:
# Runs antivirus locally ''&larr; report these results in build notes''
# Runs rsync in dry-run mode

This builder is triggered automatically after the updates builder.

If a long time has passed since the first run, this builder should be run a second time before pushing files to mirrors by hitting "Rebuild" button or forcing the builder with 2 properties set:
* '''Property 1 name''': script_repo_revision use FIREFOX_*_RELEASE tag here
* '''Property 2 name''' -> release_config: mozilla/release-firefox-mozilla-2.0.py

No reconfig needed for this builder in manual mode. Automatic mode requires reconfig only on version bump.

= Push to mirrors =

**only applicable for non release promotion releases**

For GA release builds, this step is not run automatically, and must be manually triggered (below) after receiving the go from RelMgmt. For all other releases, this step is run automatically.

Code: [http://hg.mozilla.org/build/tools/file/tip/scripts/release/stage-tasks.py scripts/release/stage-tasks.py]

* Release
** [http://buildbot-master70.bb.releng.use1.mozilla.com:8001/builders/release-mozilla-release-fennec_push_to_mirrors Fennec Release]
*** prerequisites: wait for driver email subj like '(Android|fennec|mobile firefox) $version $buildnum (uploaded|pushed|published) to (Google Play|play store)' 
*** Note: push to mirrors step can happen after we push to play store, since we serve updates through google play and not balrog (so cdn/mirrors don't need apks).
** [http://buildbot-master70.bb.releng.use1.mozilla.com:8001/builders/release-comm-esr38-thunderbird_push_to_mirrors Thunderbird 38 Release]
** [http://buildbot-master70.bb.releng.use1.mozilla.com:8001/builders/release-comm-esr45-thunderbird_push_to_mirrors Thunderbird 45 Release]
** [http://buildbot-master70.bb.releng.use1.mozilla.com:8001/builders/release-comm-esr52-thunderbird_push_to_mirrors Thunderbird 52 Release]
*** prerequisites: wait for driver email subj like 'push Thunderbird $version $buildnum to (release-cdntest channel|cdn|mirrors)'
*** Note: update_verify automation emails should be completed

This builder:
# Runs rsync and pushes files to the release directory
# Triggers mirror uptake monitoring poller.
# If partner builds are enabled (ie Firefox on release), copies some builds into place

When enableAutomaticPushToMirrors is True, this is triggered automatically after the antivirus scan is done. When False, it must be manually triggered with Force Build. Step will automatically send email when complete.

= Final Verification =

**release promotion doesn't use ReleaseFinalVerification factory. it uses taskcluster and calls final-verification.sh directly**

Code:
* 'ReleaseFinalVerification' from [http://hg.mozilla.org/build/buildbotcustom/file/tip/process/factory.py process/factory.py]
* [http://hg.mozilla.org/build/tools/file/tip/release/final-verification.sh final-verification.sh]

Once mirror uptake has reached 50 or 60% this builder should be automatically started by the uptake monitoring process. If these don't get fired automatically, manually verify uptake on bouncer, then start the builder for each platform with 'Force Build' (<em>no parameters needed</em>) from the Buildbot waterfall.  If the buildbot scheduler does decide to auto-start the builds after they've been manually started, fear not, as they are pretty quick builds and won't hurt anything.

This is a simple test which is run after mirrors are propagated to ensure there's no (major) problems with them. It does the same thing as the second part of Update Verify: for every locale/platform/version combination it checks for an update snippet, and then performs an HTTP HEAD request on the MAR the snippet points to.

The step will turn red if there are http codes other than 200 or 302, or if FAIL is found in the log. If it does this, you should have a look in the log, and find out which mirror is failing. It's a good idea to keep an eye on that mirror for awhile to make sure it finishes getting all of the files.

Searching the log for errors is made easier by downloading it and running the following locally:
 grep HTTP/1 text | grep -v 302 | grep -v 200
 grep FAIL text
 grep "HTTP/1.1 4" text
 grep "HTTP/1.1 5" text

= Bouncer Submitter =
This builder is triggered automatically after tagging completes, and creates and/or updates bouncer entries used on -cdntest and live update channels, and bouncer entries that Bedrock points at.

This step adds adds bouncer entries for installers, complete and partial updates for all platforms listed in enUSPlatforms and l10nPlatforms and locales specified in shipped-locales (en-US is being added in any case).

The step will turn red if tuxedo-add.py exits non zero. There are 2 major error sources:
* tuxedoUsername has not enough privileges to add entries
* Tuxedo is not aware of new locales added in the current version. Usually the log contains "HTTPError: HTTP Error 400: BAD REQUEST". See [[Releases/Firefox_4.0b7/BuildNotes#Update_Bouncer|this link]] for an example workflow on how to resolve this problem.

= Publish in Balrog =

**only applicable for non release promotion releases**

== Betas/Final Releases ==
'''These instructions cover _typical_ cases. If any special requirements have been requested (ie, whatsnew page) you may need to do different or additional things at the same time.'''<br/>

Before publishing, you should verify that the throttle rate is set as expected - automation will not update this for you. Compare the rate in the push e-mail (eg: 25% or 100%) to the rate on [https://aus4-admin.mozilla.org/rules the Balrog admin interface]. More on throttle rates can be found in the [[#Throttling | Throttling section of this page]]

We have a builder that updates the right parts of Balrog to push a release live. When instructed, you can use "force build" to start it.

'''NOTE:''' this step requires a "double authorization" - appropriate sign off by QE or RelMan, ''and'' verifying that automation is ready (e.g. 'ready to ship' notification).

Example buildbot URLs for the builder:
* [http://buildbot-master70.bb.releng.use1.mozilla.com:8001/builders/release-comm-beta-update_shipping_beta Thunderbird Beta]
* [http://buildbot-master70.bb.releng.use1.mozilla.com:8001/builders/release-comm-esr45-update_shipping_release Thunderbird 45 Release]
* [http://buildbot-master70.bb.releng.use1.mozilla.com:8001/builders/release-comm-esr52-update_shipping_release Thunderbird 52 Release]



''Note that the master you force this on, is not necessarily the master that takes the job, watch your e-mail for completion status. Subject will be "[https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/release_templates/update_shipping_success %(releaseName)s: Updates available on %(releaseChannel)s]".''

'''In order to guard against accidents, this builder aborts unless a special property is set. You must enter a "shipit" property on the form, whose value is also "shipit".'''

= Throttling =
The throttle rate is number from 0 to 100, used in the sense of a percentage a gas pedal is pushed down. So 
* 0 means stopped, no requests will be served an update
* 100 means full speed, all requests will be served an update
* 25 means quarter speed, 1 in 4 requests will be served an update (on average!)

Update queries which carry the parameter force=1, eg:
 https://aus3.mozilla.org/Firefox/.../update.xml?force=1
will not be throttled. Checks which are initiated by the user have this parameter, while background checks (initiated by the app itself) do not.

== Do I need to throttle? ==
Generally, there's two different ways we do throttling for releases. Most of the time we ship a release partly throttled (10% served), throttle it completely 48 hours later (at the request of RelMan through release-drivers and/or a bug), once some amount of users have it in their hands.  In certain cases (for example, when our release day coincides with a Microsoft patch Tuesday), we ship a release fully throttled (0% of requests are served an update).

'''In both cases you must make throttling changes before making the updates live.'''

We're usually told explicity "please push release xxxx fully throttled" or "please push release xxxx partly throttled", but if you're unclear about whether a release should be throttled or not, ask on release-drivers prior making the updates live. If you're confused about anything else, ask another RelEng person.

== How to throttle ==
Throttling is an attribute on each rule in Balrog. To change the throttle rate, simply adjust the "Rate" on the appropriate rule and save the changes:
[[File:Throttling in Balrog.png|upright]]

== Verifying Throttling ==
See http://people.mozilla.com/~nthomas/update-watch/release/ for example AUS links

Some links to check:
* Automatic (idle time check) update links:
** 10.0.2 en-US [https://aus3.mozilla.org/update/1/Firefox/10.0.2/20120215223356/WINNT_x86-msvc/en-US/release/update.xml win32] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Linux_x86-gcc3/en-US/release/update.xml linux] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/update.xml mac] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Linux_x86_64-gcc3/en-US/release/update.xml linux64]
** 9.0 en-US [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/WINNT_x86-msvc/en-US/release/update.xml win32] [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/Linux_x86-gcc3/en-US/release/update.xml linux] [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/update.xml mac] [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/Linux_x86_64-gcc3/en-US/release/update.xml linux64]
** 10.0.2esr en-US [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/WINNT_x86-msvc/en-US/esr/update.xml win32] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/Linux_x86-gcc3/en-US/esr/update.xml linux] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/esr/update.xml mac] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/Linux_x86_64-gcc3/en-US/esr/update.xml linux64]
** 10.0esr en-US [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/WINNT_x86-msvc/en-US/esr/update.xml win32] [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/Linux_x86-gcc3/en-US/esr/update.xml linux] [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/esr/update.xml mac] [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/Linux_x86_64-gcc3/en-US/esr/update.xml linux64]
** 3.6.27 en-US [https://aus2.mozilla.org/update/1/Firefox/3.6.27/20120216013254/WINNT_x86-msvc/en-US/release/update.xml win32] [https://aus2.mozilla.org/update/1/Firefox/3.6.27/20120216013254/Linux_x86-gcc3/en-US/release/update.xml linux] [https://aus2.mozilla.org/update/1/Firefox/3.6.27/20120216013254/Darwin_x86-gcc3-u-ppc-i386/en-US/release/update.xml mac] &larr; ''only updates to 12.0''
* Manual update links:
** 10.0.2 en-US [https://aus3.mozilla.org/update/1/Firefox/10.0.2/20120215223356/WINNT_x86-msvc/en-US/release/update.xml?force=1 win32] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Linux_x86-gcc3/en-US/release/update.xml?force=1 linux] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/update.xml?force=1 mac] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Linux_x86_64-gcc3/en-US/release/update.xml?force=1 linux64]
** 9.0 en-US [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/WINNT_x86-msvc/en-US/release/update.xml?force=1 win32] [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/Linux_x86-gcc3/en-US/release/update.xml?force=1 linux] [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/update.xml?force=1 mac] [https://aus2.mozilla.org/update/1/Firefox/9.0/20111216140209/Linux_x86_64-gcc3/en-US/release/update.xml?force=1 linux64]
** 10.0.2esr en-US [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/WINNT_x86-msvc/en-US/esr/update.xml?force=1 win32] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/Linux_x86-gcc3/en-US/esr/update.xml?force=1 linux] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/esr/update.xml?force=1 mac] [https://aus2.mozilla.org/update/1/Firefox/10.0.2/20120216092139/Linux_x86_64-gcc3/en-US/esr/update.xml?force=1 linux64]
** 10.0esr en-US [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/WINNT_x86-msvc/en-US/esr/update.xml?force=1 win32] [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/Linux_x86-gcc3/en-US/esr/update.xml?force=1 linux] [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/esr/update.xml?force=1 mac] [https://aus2.mozilla.org/update/1/Firefox/10.0/20120130064731/Linux_x86_64-gcc3/en-US/esr/update.xml?force=1 linux64]
** 3.6.27 en-US [https://aus2.mozilla.org/update/1/Firefox/3.6.27/20120216013254/WINNT_x86-msvc/en-US/release/update.xml?force=1 win32] [https://aus2.mozilla.org/update/1/Firefox/3.6.27/20120216013254/Linux_x86-gcc3/en-US/release/update.xml?force=1 linux] [https://aus2.mozilla.org/update/1/Firefox/3.6.27/20120216013254/Darwin_x86-gcc3-u-ppc-i386/en-US/release/update.xml?force=1 mac] &larr; ''only updates to 12.0''

= Post-release tasks =

'''''only applicable for non release promotion releases'''''

Thunderbird is the only type of release that is non-release-promotion. After release is live, two more tasks need to be performed: marking them as shipped and run the BB post-release builder. They should be run at the same time, after release is live.

== Mark as shipped on Ship It ==
Releases need to be marked as shipped on Ship It. To do so, visit https://ship-it.mozilla.org/releases.html, find the release in question, and click the "Shipped!" button.

== postrelease builder ==
The "post release" builder needs to be started with "force build". This builder will update bouncer aliases.

Example buildbot URLs for the builder:
* Beta
** [http://buildbot-master82.build.mozilla.org:8001/builders/release-comm-beta-thunderbird_postrelease Thunderbird Beta]

* Release
** [http://buildbot-master82.build.mozilla.org:8001/builders/release-comm-esr52-thunderbird_postrelease Thunderbird 52 Release]
''Note that the master you force this on, is not necessarily the master that takes the job, watch your e-mail for completion status. Subject will be "[https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/release_templates/default_success %(releaseName)s: completed %(stage)s]".''