Release:Release Automation on Mercurial:Updates through Shipping: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
Line 144: Line 144:
sftp -oBatchMode=no -b ~/$product-$version-build$build.batch wwdr_mac_team@privftp.apple.com:Mozilla
sftp -oBatchMode=no -b ~/$product-$version-build$build.batch wwdr_mac_team@privftp.apple.com:Mozilla
</pre>
</pre>
This script takes some time to upload all the binaries, running it in a screen might be a good idea.


The credentials can be found in the private releng repo in <tt>passwords/apple-sftp.txt.gpg</tt>.
The credentials can be found in the private releng repo in <tt>passwords/apple-sftp.txt.gpg</tt>.

Revision as of 15:01, 7 October 2014

<< Documentation

Update/snippet Generation

Deliverables:

  • Updated patcher config file
  • Update snippets on test & live channels
  • Snippet comparison log

After checking out the necessary modules and tools this builder goes to work doing the following:

  • Bumping & checking in a patcher config file
  • Creating & uploading update snippets
  • When applicable, creating snippets for previous builds of the release
  • Pushing test snippets live
  • When applicable, doing a comparison of live channel snippets vs. releasetest

This step is generally pretty failsafe but in the event that it needs to be run a second time or restarted at later step it's necessary to comment out the patcher config bump step (see below for details on how). Until bug 466999 FIXED is resolved the patcher config file can only be bumped once.

Note that both the sourceRepo and the tools repo will be updated to the patcherToolsTag. If the update-packaging tools or patcher config script has been updated since the last release it is a good idea (and sometimes necessary) to create a new UPDATE_PACKAGING_R# tag before starting the release. ReleaseEngineering/PatcherTags documents the tag history, please keep it up to date.

Snippet Comparison Log Analysis

Once updates is completed there will be a log from the snippet comparison step. This step compares the contents of the releasetest channel snippets to the contents of the live channel (beta or release) snippets. There's a few things to note about this log:

  • It will usually contain warnings about missing snippets for alphas and betas. These are expected and ignorable.
  • When there are warnings about other snippets missing, a 'Warnings' log will be created and the step will be orange. These must be investigated and explained/fixed before any live snippets are pushed.
  • If any snippets differ in contents between the test and live channels a 'Diffs' log will be created and the step will be orange. The Diffs log will list the files which differ; the full diffs can be viewed in the stdio log. Any differences must be investigated and explained/fixed before any live snippets are pushed.

Generate RC to b1 snippets

Until bug 908134 yields full integration of release RC builds on the beta channel, we need some manual work for beta1 of a new cycle. We copy some snippets to provide a complete update from X-1.0build1, X-1.0build2,... up to the new X.0beta1. QA often uses the terminology 'release candidate' or RC for the X-1.0buildN builds.

See these previous examples for what to do 32.0b1, 31.0b1. You will need to update some versions and buildIDs, ask nthomas/bhearsum/rail if you need support.

Update Verify

Code used:

Deliverables:

  • Update verify logs per platform. Details on them are below.

Update verify, in its current form, verifies a few different things:

  • That the "updater" binary works
  • That all MAR files can be applied, and result in a functionally equivalent build to the installer.
  • That all previous versions (across all locales), have snippets available

It does this in two different types of tests:

  • Full update verify unpacks a previous version's installer, pings AUS, downloads the MAR it finds, applies it, and diffs the result against the current version's installer. Differences between the two are reported in the log.
    • Run for all locales in the oldVersion
    • Run for select locales ("de", "en-US", "ru" at the time of writing) for all other previous versions
    • Completes and partials (if present) are both tested
  • Quick update verify does a subset of the above: it pings AUS, downloads the MAR it finds, and ensures that it is non-zero in size
    • Run against previous versions for all locales not tested as part of the full update verify.

Update verify is split into multiple chunks per platform (10, by default). The full and quick checks are spaced evenly throughout the chunks giving each one roughly equivalent run time.

Log Analysis

This step is pretty good at reporting errors. Most errors will turn the step red, and be pretty clearly visible in the log. Nonetheless it's a good idea to have a scan through the log for anything abnormal (searching for FAIL and WARN are helpful here).

Here's some common failures:

  • "Binary files source/bin/freebl3.chk and target/bin/freebl3.chk differ" (or softokn3.chk) on win32 in the complete MAR
    • This is normal in the win32 logs and is caused by bug 404340. Once this bug is fixed it should go away.
  • "CRC check failed" and/or "LoadSourceFile failed" on a partial MAR
    • This typically happens when the previous release's complete MAR differs from the complete package. Because partial MARs are generated by diffing the previous complete MAR with the current complete MAR the partial will fail the CRC check when this happens. There's nothing that can be done about this, because the problem is with an already-release-build. The correct thing to do here is inform release-drivers that there will not be partials available on this platform, trim them out of the candidates directory, and remove the partial snippets.
  • "FAIL: partial from xxxxx wrong size", "FAIL: actual size: 0"
    • This is usually caused by missing MARs. You should first check to see if the MAR in question exists in the candidates directory. If it doesn't, go back and look at the logs to see if there was a problem generating or uploading it. If it does exist in the candidates directory make sure the permissions are set properly (ffxbld:firefox, 664).
  • "FAIL: Could not download source xxx"
    • Usually indicates missing packages or bad permissions in the candidates directory. Take the same action as above.
  • "FAIL: no partial update found for xxx" or "FAIL: no complete update found for xxx"
    • Check the logs from the previous step to make sure 'pushsnip' was run successfully. This error usually indicates that snippets weren't pushed.

Check Permissions

This builder:

  1. Checks if the FTP directories and files have proper permissions and modes
  2. Runs rsync in dry-run mode

This builder is triggered automatically after the updates builder.

No reconfig needed for this builder in manual mode. Automatic mode requires reconfig only on version bump.

Antivirus check

This builder:

  1. Runs antivirus locally ← report these results in build notes
  2. Runs rsync in dry-run mode

This builder is triggered automatically after the updates builder.

If a long time has passed since the first run, this builder should be run a second time before pushing files to mirrors by hitting "Rebuild" button or forcing the builder with 2 properties set:

  • Property 1 name: script_repo_revision use FIREFOX_*_RELEASE tag here
  • Property 2 name -> release_config: mozilla/release-firefox-mozilla-2.0.py

No reconfig needed for this builder in manual mode. Automatic mode requires reconfig only on version bump.

Push to mirrors

For GA release builds, this step is not run automatically, and must be manually triggered (below) after receiving the go from RelMgmt. For all other releases, this step is run automatically.

Code: scripts/release/stage-tasks.py

This builder:

  1. Runs rsync and pushes files to the release directory
  2. Triggers mirror uptake monitoring poller.

When enableAutomaticPushToMirrors is True, this is triggered automatically after the antivirus scan is done. When False, it must be manually triggered with Force Build. Step will automatically send email when complete.

Final Verification

Code:

Once mirror uptake has reached 50 or 60% this builder should be automatically started by the uptake monitoring process. If these don't get fired automatically, manually verify uptake on bouncer, then start the builder for each platform with 'Force Build' (no parameters needed) from the Buildbot waterfall. If the buildbot scheduler does decide to auto-start the builds after they've been manually started, fear not, as they are pretty quick builds and won't hurt anything.

This is a simple test which is run after mirrors are propagated to ensure there's no (major) problems with them. It does the same thing as the second part of Update Verify: for every locale/platform/version combination it checks for an update snippet, and then performs an HTTP HEAD request on the MAR the snippet points to.

The step will turn red if there are http codes other than 200 or 302, or if FAIL is found in the log. If it does this, you should have a look in the log, and find out which mirror is failing. It's a good idea to keep an eye on that mirror for awhile to make sure it finishes getting all of the files.

Searching the log for errors is made easier by downloading it and running the following locally: 

grep HTTP/1 text | grep -v 302 | grep -v 200
grep FAIL text
grep "HTTP/1.1 4" text
grep "HTTP/1.1 5" text

Upload files to Apple for Whitelisting

This is a temporary measured, only needed until bug 1046306 is fixed. Firefox 33 is the last release this should be needed for. This step only needs to happen for beta 1's, release channel, and ESRs.

As ffxbld@upload1.dmz.scl3.mozilla.com (from any master): 

ssh -i .ssh/ffxbld_dsa ffxbld@stage.mozilla.org

product=foo
version=x.y
build=z
wget -O create-apple-sftp-batchfile.py https://hg.mozilla.org/build/braindump/raw-file/default/releases-related/create-apple-sftp-batchfile.py
cd /pub/mozilla.org/$product/candidates
python ~/create-apple-sftp-batchfile.py $version-candidates | grep build$build | tee ~/$product-$version-build$build.batch
sftp -oBatchMode=no -b ~/$product-$version-build$build.batch wwdr_mac_team@privftp.apple.com:Mozilla

This script takes some time to upload all the binaries, running it in a screen might be a good idea.

The credentials can be found in the private releng repo in passwords/apple-sftp.txt.gpg.

Apple needs to be notified after all releases (usually release channel + ESR31 + maybe Thunderbird) are uploaded so they can start processing them. Contact Ben Hearsum or Lawrence Mandel to have this done. INCLUDE THE DIRECTORIES ON SFTP server containing the new uploads in the email. Lawrence needs to send that list to Apple.

Bouncer Submitter

Code:

Configuration:

  • firefox-tuxedo.ini for stable versions, and firefox-devpreview-tuxedo.ini for Alpha releases.
  • BuildSlaves.py file should contain the following variables:
    • tuxedoUsername: Tuxedo username with appropriate permissions for adding new entries via API
    • tuxedoPassword: password for this user

This builder can be started any time with 'Force Build' from the Buildbot waterfall.

This step adds adds bouncer entries for installers, complete and partial updates for all platforms listed in enUSPlatforms and l10nPlatforms and locales specified in shipped-locales (en-US is being added in any case).

The step will turn red if tuxedo-add.py exits non zero. There are 2 major error sources:

  • tuxedoUsername has not enough privileges to add entries
  • Tuxedo is not aware of new locales added in the current version. Usually the log contains "HTTPError: HTTP Error 400: BAD REQUEST". See this link for an example workflow on how to resolve this problem.

Push snippets

This happens after QA signs off and we get an explicit "go" from Release Management. Do not push snippets unless you're certain you should. If you're unsure, ask someone. NOTE: for all beta releases except b1, we push snippets on QA signoff.

  • For example:
# As ffxbld or tbirdbld@aus3-staging.mozilla.org
# You probably want to use screen!!
cd /opt/aus2/snippets/staging
~/bin/pushsnip Firefox-13.0-build1
  • If appropriate, publish to balrog, then
  • Send email drivers list that release is live

Publish in Balrog

We have a builder that updates the right parts of Balrog to push a release live. When instructed, you can use "force build" to start it. (Balrog replaces pushing snippets, so should be done at the same time.) Example buildbot URLs for the builder:

Note that the master you force this on, is not necessarily the master that takes the job, watch your e-mail for completion status

In order to guard against accidents, this builder aborts unless a special property is set. You must enter a "shipit" property on the form, whose value is also "shipit".

Throttling

AUS2 supports two types of throttling, which are applied in this order of precedence

  • app-version-channel specific
  • global

We don't use the global throttle, so this document will not talk about it specifically.

In both cases, an update query which carries the parameter force=1, eg 

https://aus3.mozilla.org/Firefox/.../update.xml?force=1

will not be throttled. Checks which are initiated by the user have this parameter, while background checks (initiated by the app itself) do not.

In both cases the throttle is number from 0 to 100, used in the sense of a percentage a gas pedal is pushed down. So

  • 0 means stopped, no requests will be served an update
  • 100 means full speed, all requests will be served an update
  • 25 means quarter speed, 1 in 4 requests will be served an update (on average!)

These are configured in config-dist.php. The machines that serve AUS symlink/copy config-dist.php to config.php.

Configuring Throttling

App-version-channel throttling allows you to configure a throttle level for an application and version, and to set exceptions for channels that it should not apply to. The version comes from the request, rather than the update served for that request.

eg, Updates from Firefox 3.6.17 are turned off on the release channel, but not beta or either test channel 

$productThrottling = array(
    'Firefox' => array(
        '3.6.17' => 0
    )
);
$throttleExceptions = array(
    '3.6.17' => array (
        'betatest',
        'releasetest',
        'beta'
    )
);

We always add exceptions for the two test channels, so that QA and RelEng can test updates without any randomness.

Do I need to throttle?

Generally, there's two different ways we do throttling for releases. Most of the time we ship a release partly throttled (10% served), throttle it completely 48 hours later (at the request of RelMan through release-drivers and/or a bug), once some amount of users have it in their hands. In certain cases (for example, when our release day coincides with a Microsoft patch Tuesday), we ship a release fully throttled (0% of requests are served an update).

In both cases you must make throttling changes before pushing the snippets.

We're usually told explicity "please push release xxxx fully throttled" or "please push release xxxx partly throttled", but if you're unclear about whether a release should be throttled or not, ask on release-drivers prior to pushing snippets. If you're confused about anything else, ask another RelEng person.

How to throttle

Throttling is a multi step process that involves:

  1. Changing the AUS config (and getting review on it)
  2. Moving the AUS2_PRODUCTION tag on config-dist.php
    • move tag only on config-dist.php: "cvs tag -F AUS2_PRODUCTION inc/config-dist.php"
  3. Asking IT to deploy the new code (bug template).
    • include the new revision number of config-dist.php (REV_NUMBER in template IT uses a script that takes that rev number as input.) from: " cvs status -v inc/config-dist.php | grep AUS2_PRODUCTION"
  4. When the change is deployed, send email to r-d summarizing the change.

For example, when we throttled to release 23.0 this patch was used. After review it was committed to the CVS repository, and the AUS2_PRODUCTION tag was moved to the new revision. Not everyone has commit access, ask aki, bhearsum, Callek, catlee, coop, kmoir, nthomas, or rail for help. bug 901913 was filed for IT to deploy the code. Note that we file such deployment bugs as blockers, otherwise they won't happen quickly enough.

Verifying Throttling

See http://people.mozilla.com/~nthomas/update-watch/release/ for example AUS links

Some links to check:

Post-release tasks

After QA has signed off on a Desktop release, the "post release" builder needs to be started with "force build". What constitutes sign off varies by release type, see below and/or double check with qa in irc.

This builder will remove index files where they are present, update the "latest" symlink to point at the new release, and update bouncer aliases. XULRunner post release builders will be run automatically after Firefox post release.

Example buildbot URLs for the builder:

Note that the master you force this on, is not necessarily the master that takes the job, watch your e-mail for completion status

Retrieved from "https://wiki.allizom.org/index.php?title=Release:Release_Automation_on_Mercurial:Updates_through_Shipping&oldid=1022973"