Release:Release Automation on Mercurial:Updates through Shipping: Difference between revisions

Jump to navigation Jump to search

Release:Release Automation on Mercurial:Updates through Shipping (view source)

Revision as of 16:59, 11 December 2017

4,003 bytes removed ,  11 December 2017
bm70 was decomm'd
(→‎Push to mirrors: we do some partner stuff too)
(bm70 was decomm'd)
 
(83 intermediate revisions by 10 users not shown)
Line 4: Line 4:
Deliverables:
Deliverables:
* Updated patcher config file
* Updated patcher config file
* Update snippets on test & live channels
* Updated update verify configs
* Snippet comparison log
* Release blobs submitted to production Balrog
* Updated test channel rules in Balrog, pointing at the new release Blob


After checking out the necessary modules and tools this builder goes to work doing the following:
After checking out the necessary modules and tools this builder goes to work doing the following:
* Bumping & checking in a patcher config file
* Bumping & checking in a patcher config file
* Creating & uploading update snippets
* Creating & uploading update snippets
* When applicable, creating snippets for previous builds of the release
* Pushing test snippets live
* When applicable, doing a comparison of live channel snippets vs. releasetest


This step is generally pretty failsafe but in the event that it needs to be run a second time or restarted at later step it's necessary to comment out the patcher config bump step (see [[#Restarting_a_builder_from_a_certain_point | below]] for details on how). Until <strike>{{bug|466999}} FIXED</strike> is resolved the patcher config file can only be bumped once.
= Set-up whatsnew page =
At the time of writing, manual work is required if we're shipping a release that has a mix of old versions that need/don't need a whatsnew page. If Release Management hasn't specified that a whatsnew page is needed, there's nothing to do. This must be done after initial update submission is done, and you'll need to know which where the cut off line for the whatsnew is. For example, when 35.0 shipped everything earlier than 34.0 required a whatsnew page, but 34.0 and 34.0.5 did not. If you don't grok how Balrog rules work it would be a good idea [[Balrog#Rules | to read up on them]] before getting started here.


Note that both the sourceRepo and the tools repo will be updated to the patcherToolsTag. If the update-packaging tools or patcher config script has been updated since the last release it is a good idea (and sometimes necessary) to create a new UPDATE_PACKAGING_R# tag before starting the release. [[ReleaseEngineering/PatcherTags]] documents the tag history, please keep it up to date.
Note: do not modify blobs until both "updates" builders have finished. Your changes may be overwritten otherwise.


== Snippet Comparison Log Analysis ==
To create a blob with the whatsnew page enabled, do the following:
Once updates is completed there will be a log from the snippet comparison step. This step compares the contents of the releasetest channel snippets to the contents of the live channel (beta or release) snippets. There's a few things to note about this log:
* Load the [https://aus4-admin.mozilla.org/ui/ Balrog Admin UI].
* It will usually contain warnings about missing snippets for alphas and betas. These are expected and ignorable.
* Go to the "Releases" section.
* When there are warnings about other snippets missing, a 'Warnings' log will be created and the step will be orange. These must be investigated and explained/fixed before any live snippets are pushed.
* Find the release you're working on by filtering.
* If any snippets differ in contents between the test and live channels a 'Diffs' log will be created and the step will be orange. The Diffs log will list the files which differ; the full diffs can be viewed in the stdio log. Any differences must be investigated and explained/fixed before any live snippets are pushed.
* Click "Download".
* Edit the local file and add the following at the top level of the blob (replace the VERSION variable with the version of the release, do NOT replace %LOCALE% or %OLD_VERSION% - those are substituted by Balrog/Firefox):
"openURL": "https://www.mozilla.org/%LOCALE%/firefox/$MANUALLY_REPLACE_THIS_VARIABLE_CALLED_VERSION/whatsnew/?oldversion=%OLD_VERSION%",
"actions": "showURL",
* Change the "name" in the blob to something unique (usually by appending "-whatsnew" to it).
* Back in the Admin UI, click "Add a new release".
* The name should match the one you just edited in the blob, product should be Firefox.
* Upload the new blob.
 
Now that you've got the the new blob you need to adjust rules to point at it. This is usually done in two stage: once for the test channels, and once for the live channel. Test channel updates are typically done as soon as the automated channel updates become available. Live channel updates typically happen at the same time we push the release live. Take care when modifying rules - any changes you make to live channels take effect immediately.
 
Because the changes that need to be made are highly dependent on the state of the rules at the time and the specific versions requiring/not requiring a whatsnew page, specific step-by-step instructions are explicitly not provided. You may need to manipulate existing rules, add new ones, or even delete some. As an example, here's what the release live and test channels rules looked like after 35.0 was shipped:<br/>
[[File:Balrog_whatsnew_rules_example.png]]
 
The rules above are listed from highest priority to lowest. The first is ignorable for our purposes (it's a long standing watershed update rule for users on versions earlier than 10.0). The next 3 are are rules that point to the whatsnew version of the blob created above - one rule each for the cdntest, localtest, and live channels. The last 3 point users at the non-whatsnew blob that was fully created by automation. Note that the whatsnew rules specify a version ("<34.0") and have a higher priority.


= Update Verify =
= Update Verify =
Line 62: Line 75:


= Check Permissions =
= Check Permissions =
**only applicable for non release promotion releases**
This builder:
This builder:
# Checks if the FTP directories and files have proper permissions and modes
# Checks if the FTP directories and files have proper permissions and modes
Line 71: Line 85:


= Antivirus check =
= Antivirus check =
 
**only applicable for non release promotion releases. release promotion bundles antivirus within beetmover tasks**
This builder:
This builder:
# Runs antivirus locally ''&larr; report these results in build notes''
# Runs antivirus locally ''&larr; report these results in build notes''
Line 85: Line 99:


= Push to mirrors =
= Push to mirrors =
'''''only applicable for non release promotion releases'''''
For GA release builds, this step is not run automatically, and must be manually triggered (below) after receiving the go from RelMgmt. For all other releases, this step is run automatically.
For GA release builds, this step is not run automatically, and must be manually triggered (below) after receiving the go from RelMgmt. For all other releases, this step is run automatically.


Code: [http://hg.mozilla.org/build/tools/file/tip/scripts/release/stage-tasks.py scripts/release/stage-tasks.py]


* Release
* Release
** [http://buildbot-master70.srv.releng.use1.mozilla.com:8001/builders/release-mozilla-release-firefox_push_to_mirrors Firefox Release]
** [http://buildbot-master71.bb.releng.use1.mozilla.com:8001/builders/release-comm-esr52-thunderbird_push_to_mirrors Thunderbird 52 Release]
** [http://buildbot-master70.srv.releng.use1.mozilla.com:8001/builders/release-mozilla-release-fennec_push_to_mirrors Fennec Release]
*** prerequisites: wait for driver email subj like 'push Thunderbird $version $buildnum to (release-cdntest channel|cdn|mirrors)'
** [http://buildbot-master70.srv.releng.use1.mozilla.com:8001/builders/release-comm-esr31-thunderbird_push_to_mirrors Thunderbird 31 Release]
*** Note: update_verify automation emails should be completed
 
* ESR31
** [http://buildbot-master70.srv.releng.use1.mozilla.com:8001/builders/release-mozilla-esr31-firefox_push_to_mirrors Firefox ESR31]
** [http://buildbot-master70.srv.releng.use1.mozilla.com:8001/builders/release-mozilla-esr31-fennec_push_to_mirrors Fennec ESR31]


This builder:
This builder:
Line 104: Line 116:


When enableAutomaticPushToMirrors is True, this is triggered automatically after the antivirus scan is done. When False, it must be manually triggered with Force Build. Step will automatically send email when complete.
When enableAutomaticPushToMirrors is True, this is triggered automatically after the antivirus scan is done. When False, it must be manually triggered with Force Build. Step will automatically send email when complete.
The code used to be in
[http://hg.mozilla.org/build/tools/file/tip/scripts/release/stage-tasks.py scripts/release/stage-tasks.py] (for historical revisions) but is now a [https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/release/push-candidate-to-releases.py?q=path%3Apush-candidate-to-releases.py&redirect_type=single mozharness] script


= Final Verification =
= Final Verification =
**release promotion doesn't use ReleaseFinalVerification factory. it uses taskcluster and calls final-verification.sh directly**
Code:
Code:
* 'ReleaseFinalVerification' from [http://hg.mozilla.org/build/buildbotcustom/file/tip/process/factory.py process/factory.py]
* 'ReleaseFinalVerification' from [http://hg.mozilla.org/build/buildbotcustom/file/tip/process/factory.py process/factory.py]
Line 121: Line 139:
  grep "HTTP/1.1 4" text
  grep "HTTP/1.1 4" text
  grep "HTTP/1.1 5" text
  grep "HTTP/1.1 5" text
= Upload files to Apple for Whitelisting =
(requires: completed partner builds, if any)
This is a temporary measure, only needed until {{bug|1046306}} is fixed. Firefox 33 is the last release this is needed for, so it only ESRs need to be uploaded.
As ffxbld@upload1.dmz.scl3.mozilla.com (from any master):
ssh -i .ssh/ffxbld_dsa ffxbld@stage.mozilla.org
<pre>
product=foo
version=x.y
build=z
wget -O create-apple-sftp-batchfile.py https://hg.mozilla.org/build/braindump/raw-file/default/releases-related/create-apple-sftp-batchfile.py
cd /pub/mozilla.org/$product/candidates
if [[ "$build" == "1" ]]; then
  python ~/create-apple-sftp-batchfile.py $version-candidates | tee ~/$product-$version-build$build.batch
else
  python ~/create-apple-sftp-batchfile.py $version-candidates | grep build$build | tee ~/$product-$version-build$build.batch
fi
sftp -oBatchMode=no -b ~/$product-$version-build$build.batch wwdr_mac_team@privftp.apple.com:Mozilla
</pre>
This script takes some time to upload all the binaries, running it in a screen might be a good idea.
The credentials can be found in the private releng repo in <tt>passwords/apple-sftp.txt.gpg</tt>.
Apple needs to be notified after all releases (usually release channel + ESR31 + maybe Thunderbird) are uploaded so they can start processing them. Contact Ben Hearsum or Lawrence Mandel to have this done. '''INCLUDE THE DIRECTORIES ON SFTP server containing the new uploads''' in the email. Lawrence needs to send that list to Apple.


= Bouncer Submitter =
= Bouncer Submitter =
Code:
This builder is triggered automatically after tagging completes, and creates and/or updates bouncer entries used on -cdntest and live update channels, and bouncer entries that Bedrock points at.
* 'TuxedoEntrySubmitterFactory' from [http://hg.mozilla.org/build/buildbotcustom/file/tip/process/factory.py process/factory.py]
* [http://hg.mozilla.org/build/tools/file/tip/release/tuxedo-add.py tuxedo-add.py]
Configuration:
* [http://hg.mozilla.org/build/tools/file/tip/release/firefox-tuxedo.ini firefox-tuxedo.ini] for stable versions, and [http://hg.mozilla.org/build/tools/file/tip/release/firefox-devpreview-tuxedo.ini firefox-devpreview-tuxedo.ini] for Alpha releases.
* BuildSlaves.py file should contain the following variables:
** tuxedoUsername: Tuxedo username with appropriate permissions for adding new entries via API
** tuxedoPassword: password for this user
 
This builder can be started any time with 'Force Build' from the Buildbot waterfall.


This step adds adds bouncer entries for installers, complete and partial updates for all platforms listed in enUSPlatforms and l10nPlatforms and locales specified in shipped-locales (en-US is being added in any case).
This step adds adds bouncer entries for installers, complete and partial updates for all platforms listed in enUSPlatforms and l10nPlatforms and locales specified in shipped-locales (en-US is being added in any case).
Line 169: Line 149:
* Tuxedo is not aware of new locales added in the current version. Usually the log contains "HTTPError: HTTP Error 400: BAD REQUEST". See [[Releases/Firefox_4.0b7/BuildNotes#Update_Bouncer|this link]] for an example workflow on how to resolve this problem.
* Tuxedo is not aware of new locales added in the current version. Usually the log contains "HTTPError: HTTP Error 400: BAD REQUEST". See [[Releases/Firefox_4.0b7/BuildNotes#Update_Bouncer|this link]] for an example workflow on how to resolve this problem.


= Push snippets =
= Publish in Balrog =
This happens after QA signs off and we get an explicit "go" from Release Management. Do not push snippets unless you're certain you should. If you're unsure, ask someone. '''''NOTE:''' for all Fx beta releases except b1, we push snippets on QA signoff. (TB always require relman signoff)''
 
'''''only applicable for non release promotion releases'''''
 
== Betas/Final Releases ==
'''These instructions cover _typical_ cases. If any special requirements have been requested (ie, whatsnew page) you may need to do different or additional things at the same time.'''<br/>
 
Before publishing, you should verify that the throttle rate is set as expected - automation will not update this for you. Compare the rate in the push e-mail (eg: 25% or 100%) to the rate on [https://aus4-admin.mozilla.org/rules the Balrog admin interface]. More on throttle rates can be found in the [[#Throttling | Throttling section of this page]]


* For example:
We have a builder that updates the right parts of Balrog to push a release live. When instructed, you can use "force build" to start it.
# As ffxbld or tbirdbld@aus3-staging.mozilla.org
# You probably want to use screen!!
cd /opt/aus2/snippets/staging
~/bin/pushsnip Firefox-13.0-build1


* If appropriate, publish to balrog, then
'''NOTE:''' this step requires a "double authorization" - appropriate sign off by QE or RelMan, ''and'' verifying that automation is ready (e.g. 'ready to ship' notification).
* Send email drivers list that release is live


= Publish in Balrog =
We have a builder that updates the right parts of Balrog to push a release live. When instructed, you can use "force build" to start it. (Balrog replaces pushing snippets, so should be done at the same time.)
Example buildbot URLs for the builder:
Example buildbot URLs for the builder:
* Beta (when QA signs off and requests the snippet push):
* [http://buildbot-master71.bb.releng.use1.mozilla.com:8001/builders/release-comm-beta-update_shipping_beta Thunderbird Beta]
** [http://buildbot-master70.srv.releng.use1.mozilla.com:8001/builders/release-mozilla-beta-update_shipping Firefox Beta]
* [http://buildbot-master71.bb.releng.use1.mozilla.com:8001/builders/release-comm-esr52-update_shipping_release Thunderbird 52 Release]
** [http://buildbot-master70.srv.releng.use1.mozilla.com:8001/builders/release-comm-beta-update_shipping Thunderbird Beta]
''Note that the master you force this on, is not necessarily the master that takes the job, watch your e-mail for completion status. Subject will be "[https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/release_templates/update_shipping_success %(releaseName)s: Updates available on %(releaseChannel)s]".''


In order to guard against accidents, this builder aborts unless a special property is set. You must enter a "shipit" property on the form, whose value is also "shipit".


= Throttling =
AUS2 supports two types of throttling, which are applied in this order of precedence
* app-version-channel specific
* global


We don't use the global throttle, so this document will not talk about it specifically.
''Note that the master you force this on, is not necessarily the master that takes the job, watch your e-mail for completion status. Subject will be "[https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/release_templates/update_shipping_success %(releaseName)s: Updates available on %(releaseChannel)s]".''


In both cases, an update query which carries the parameter force=1, eg
'''In order to guard against accidents, this builder aborts unless a special property is set. You must enter a "shipit" property on the form, whose value is also "shipit".'''
https://aus3.mozilla.org/Firefox/.../update.xml?force=1
will not be throttled. Checks which are initiated by the user have this parameter, while background checks (initiated by the app itself) do not.


In both cases the throttle is number from 0 to 100, used in the sense of a percentage a gas pedal is pushed down. So  
= Throttling =
The throttle rate is number from 0 to 100, used in the sense of a percentage a gas pedal is pushed down. So  
* 0 means stopped, no requests will be served an update
* 0 means stopped, no requests will be served an update
* 100 means full speed, all requests will be served an update
* 100 means full speed, all requests will be served an update
* 25 means quarter speed, 1 in 4 requests will be served an update (on average!)
* 25 means quarter speed, 1 in 4 requests will be served an update (on average!)


These are configured in [http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/webtools/aus/xml/inc/config-dist.php&rev=AUS2_PRODUCTION&cvsroot=/cvsroot config-dist.php]. The machines that serve AUS symlink/copy config-dist.php to config.php.
Update queries which carry the parameter force=1, eg:
 
https://aus3.mozilla.org/Firefox/.../update.xml?force=1
== Configuring Throttling ==
will not be throttled. Checks which are initiated by the user have this parameter, while background checks (initiated by the app itself) do not.
App-version-channel throttling allows you to configure a throttle level for an application and version, and to set exceptions for channels that it should not apply to. The version comes from the request, rather than the update served for that request.
 
eg, Updates from Firefox 3.6.17 are turned off on the release channel, but not beta or either test channel
$productThrottling = array(
    'Firefox' => array(
        '3.6.17' => 0
    )
);
$throttleExceptions = array(
    '3.6.17' => array (
        'betatest',
        'releasetest',
        'beta'
    )
);
 
We always add exceptions for the two test channels, so that QA and RelEng can test updates without any randomness.


== Do I need to throttle? ==
== Do I need to throttle? ==
Generally, there's two different ways we do throttling for releases. Most of the time we ship a release partly throttled (10% served), throttle it completely 48 hours later (at the request of RelMan through release-drivers and/or a bug), once some amount of users have it in their hands.  In certain cases (for example, when our release day coincides with a Microsoft patch Tuesday), we ship a release fully throttled (0% of requests are served an update).
Generally, there's two different ways we do throttling for releases. Most of the time we ship a release partly throttled (10% served), throttle it completely 48 hours later (at the request of RelMan through release-drivers and/or a bug), once some amount of users have it in their hands.  In certain cases (for example, when our release day coincides with a Microsoft patch Tuesday), we ship a release fully throttled (0% of requests are served an update).


'''In both cases you must make throttling changes before pushing the snippets.'''
'''In both cases you must make throttling changes before making the updates live.'''


We're usually told explicity "please push release xxxx fully throttled" or "please push release xxxx partly throttled", but if you're unclear about whether a release should be throttled or not, ask on release-drivers prior to pushing snippets. If you're confused about anything else, ask another RelEng person.
We're usually told explicity "please push release xxxx fully throttled" or "please push release xxxx partly throttled", but if you're unclear about whether a release should be throttled or not, ask on release-drivers prior making the updates live. If you're confused about anything else, ask another RelEng person.


== How to throttle ==
== How to throttle ==
Throttling is a multi step process that involves:
Throttling is an attribute on each rule in Balrog. To change the throttle rate, simply adjust the "Rate" on the appropriate rule and save the changes:
# [[ Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Configuring_Throttling | Changing the AUS config ]] (and getting review on it)
[[File:Throttling in Balrog.png|upright]]
# Moving the AUS2_PRODUCTION tag on config-dist.php
#* move tag '''only''' on config-dist.php: "<tt>cvs tag -F AUS2_PRODUCTION inc/config-dist.php</tt>"
# Asking IT to deploy the new code ([https://bugzilla.mozilla.org/enter_bug.cgi?assigned_to=server-ops-webops%40mozilla-org.bugs&bug_severity=blocker&bug_status=NEW&comment=fyi%2C%20instructions%20at%3A%20https%3A%2F%2Fmana.mozilla.org%2Fwiki%2Fdisplay%2Fwebsites%2Faus2.mozilla.org&component=WebOps%3A%20Product%20Delivery&form_name=enter_bug&product=Infrastructure%20%26%20Operations&rep_platform=All&short_desc=Please%20update%20AUS%20to%20AUS2_PRODUCTION%20tag%20%28REV_NUMBER%29&version=other bug template]).
#* include the new revision number of config-dist.php (''REV_NUMBER in template'' IT uses a [https://mana.mozilla.org/wiki/display/websites/aus2.mozilla.org script] that takes that rev number as input.) from: "<tt> cvs status -v inc/config-dist.php  | grep AUS2_PRODUCTION</tt>"  
# When the change is deployed, send email to r-d summarizing the change.
 
For example, when we throttled to release 23.0 [https://bug901780.bugzilla.mozilla.org/attachment.cgi?id=786107 this patch] was used. After review it was committed to the CVS repository, and the AUS2_PRODUCTION tag was moved to the new revision. Not everyone has commit access, ask aki, bhearsum, Callek, catlee, coop, kmoir, nthomas, or rail for help. {{Bug|901913}} was filed for IT to deploy the code. Note that we file such deployment bugs as blockers, otherwise they won't happen quickly enough.


== Verifying Throttling ==
== Verifying Throttling ==
Line 264: Line 211:


= Post-release tasks =
= Post-release tasks =
After QA has signed off on a release, the "post release" builder needs to be started with "force build". What constitutes sign off varies by release type, see below and/or double check  with qa in irc.


This builder will remove index files where they are present, update the "latest" symlink to point at the new release, and update bouncer aliases. XULRunner post release builders will be run automatically after Firefox post release.
'''''only applicable for non release promotion releases'''''
 
Thunderbird is the only type of release that is non-release-promotion. After release is live, two more tasks need to be performed: marking them as shipped and run the BB post-release builder. They should be run at the same time, after release is live.
 
== Mark as shipped on Ship It ==
Releases need to be marked as shipped on Ship It. To do so, visit https://ship-it.mozilla.org/releases.html, find the release in question, and click the "Shipped!" button.
 
== postrelease builder ==
The "post release" builder needs to be started with "force build". This builder will update bouncer aliases.


Example buildbot URLs for the builder:
Example buildbot URLs for the builder:
* Beta
* Beta
** [http://buildbot-master82.build.mozilla.org:8001/builders/release-mozilla-beta-firefox_postrelease Firefox Beta] (after QA signs off updates on beta)
** [http://buildbot-master82.build.mozilla.org:8001/builders/release-comm-beta-thunderbird_postrelease Thunderbird Beta]
** [http://buildbot-master82.build.mozilla.org:8001/builders/release-mozilla-beta-fennec_postrelease Fennec Beta] (after QA signs on Play Store availability)
** [http://buildbot-master82.build.mozilla.org:8001/builders/release-comm-beta-thunderbird_postrelease Thunderbird Beta] (after QA signs off updates on beta)


* Release
* Release
** [http://buildbot-master82.build.mozilla.org:8001/builders/release-mozilla-release-firefox_postrelease Firefox Release] (after QA signs off updates on release, possibly including throttling)
** [http://buildbot-master82.build.mozilla.org:8001/builders/release-comm-esr52-thunderbird_postrelease Thunderbird 52 Release]
** [http://buildbot-master82.build.mozilla.org:8001/builders/release-mozilla-release-fennec_postrelease Fennec Release] (after QA signs on Play Store availability)
** [http://buildbot-master82.build.mozilla.org:8001/builders/release-comm-esr31-thunderbird_postrelease Thunderbird 31 Release] (after QA signs off updates on release, possibly including throttling)
 
* ESR31
** [http://buildbot-master82.build.mozilla.org:8001/builders/release-mozilla-esr31-firefox_postrelease Firefox ESR31] (after QA signs off updates on esr)
** [http://buildbot-master82.build.mozilla.org:8001/builders/release-mozilla-esr31-fennec_postrelease Fennec ESR31] (after QA signs on Play Store availability)
 
''Note that the master you force this on, is not necessarily the master that takes the job, watch your e-mail for completion status. Subject will be "[https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/release_templates/default_success %(releaseName)s: completed %(stage)s]".''
''Note that the master you force this on, is not necessarily the master that takes the job, watch your e-mail for completion status. Subject will be "[https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/release_templates/default_success %(releaseName)s: completed %(stage)s]".''
Callek
Account confirmers, Anti-spam team, Confirmed users, Bureaucrats and Sysops emeriti
1,529

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/1038187...1185649"

Navigation menu