|
|
| (49 intermediate revisions by 8 users not shown) |
| Line 3: |
Line 3: |
| After we get a "go to build" e-mail from Release Management there's some manual steps to be run to prepare for and start a release. This page describes them in detail. | | After we get a "go to build" e-mail from Release Management there's some manual steps to be run to prepare for and start a release. This page describes them in detail. |
|
| |
|
| = L10N Changesets = | | = Overall beta release = |
| Currently we have separate changesets files for Fennec and Firefox. Both are generated from the [https://l10n.mozilla.org/shipping/milestones l10n dashboard]<br />
| | [[File:Beta-release-schema.png]] |
| First log in to the dashboard with your LDAP and then follow the instructions below
| |
|
| |
|
| Note: when clicking "Ship It" you may be notified that there are pending sign-offs for one or more locales. It is ok to proceed for betas. For final releases we leave the decision with Release Management.
| | Source: [[File:Beta-cycle.zip]] |
|
| |
|
| == Creating beta milestones ==
| | See also: |
| Sometimes the beta milestones that we need have not been created and we can do so if we have the right permissions:
| | * [http://moz-releng-docs.readthedocs.org/en/latest/release_workflows/fx_ga_release.html GA Workflow] (used for both GA and first Beta builds). |
| * load https://l10n.mozilla.org/shipping/milestones and sign-in | | * [http://moz-releng-docs.readthedocs.org/en/latest/release_workflows/fx_beta_release.html Beta Workflow] (used for remaining beta releases) |
| * click on the link at the bottom of the page: "You may be able to create new milestones."
| |
| * click the checkbox(es) for the product(s) you want to bump.
| |
| ** This should default to something useful, e.g. "Beta Build 3" and "fx16_beta_b3", but you can edit them if needed. | |
| * hit "submit" and the milestones will be created.
| |
|
| |
|
| ===Fennec=== | | = Submit to Ship It = |
| * Click "ship" to load up the milestone (eg: Fennec 16 Beta Build 3)
| | <b>''This task is completed by Release Management''</b> |
| ** This will take you to a page like [https://l10n.mozilla.org/shipping/confirm-ship?ms=fennec16_beta_b3 https://l10n.mozilla.org/shipping/confirm-ship?ms=fennec16_beta_b3]
| |
| * Click "ship it"
| |
| * Click "shipping tools" at the top
| |
| * Platforms should read "android"; multi-locale should read "android-multilocale".
| |
| * Click "Add"
| |
| ** repo: "releases/mozilla-beta" (you MUST use mozilla-beta, even for mozilla-release based releases)
| |
| ** branch: "default"
| |
| ** path: "mobile/android/locales/maemo-locales"
| |
| * Click "l10n-changesets.json"
| |
| * Copy and paste it into your repo's l10n-changesets_mobile-{version}.json
| |
|
| |
|
| ===Firefox or Thunderbird===
| | Releases are initiated through the [https://ship-it.mozilla.org/ Ship It webapp] (through the Mozilla VPN). When you're ready to start a release, you can use the [https://ship-it.mozilla.org/submit_release.html "Submit a new release" page] to submit it. Once it's in the system you should find '''someone else''' to review it for you. They can see it and mark it as ready on [https://ship-it.mozilla.org/releases.html the "View releases" page]. If multiple releases are expected around the same time it's often best to wait until all are ready for review, so that they can be started at the same time. |
|
| |
|
| * Click "ship" to load up the milestone (eg: Firefox 16 Beta Build 3)
| | Track Ship-It status on the [https://ship-it.mozilla.org/releases.html the "View releases" page]. If there are errors reported, check your email for details from "<tt>cltbld</tt>" with subject starting "<tt>[release-runner]</tt>". |
| ** This will take you to [https://l10n.mozilla.org/shipping/confirm-ship?ms=fx16_beta_b3 https://l10n.mozilla.org/shipping/confirm-ship?ms=fx16_beta_b3]
| |
| * Click "ship it"
| |
| ** This will take you to [https://l10n.mozilla.org/shipping/about-milestone/fx16_beta_b3 https://l10n.mozilla.org/shipping/about-milestone/fx16_beta_b3]
| |
| * Click "shipping tools" at the top
| |
| * Click "l10n-changesets"
| |
| * copy and paste the list into your changesets file (eg: l10n-changesets_mozilla-{version})
| |
|
| |
|
| = Finding Build Master = | | == Defaults == |
| Only specific build masters should be used for release builds, and these are allocated to specific branches. Find the correct choice for your builds from the [http://hg.mozilla.org/build/tools/file/tip/buildfarm/maintenance/production-masters.json production-masters.json] file. <small>([http://hg.mozilla.org/build/braindump/file/tip/releases-related/prod_master_lister.py This script] can parse the json and output what you need.)</small>
| | Unless otherwise stated by RelMan we stick to certain assumptions about releases: |
| | * Firefox betas need partial updates from the previous 2 betas |
| | * Firefox releases (that is, a release on the release channel) need partial updates from the 3 past versions with the most users. If you don't know how to get this information ask someone in RelEng or RelMan. |
| | * A "go" for Firefox implies a "go" for Fennec, too |
|
| |
|
| = Clobbering = | | = Paperwork = |
| Mark a clobberer for "Any master", $branch, "Any builder" with [https://secure.pub.build.mozilla.org/clobberer/ clobberer]. This will instruct any slave that did the last release on $branch to clobberer their release directories. Ideally, this should be done hours or more before the release starts, so that cleanup occurs before the release job starts. Clobbering isn't mandatory if your release config has base_clobber_url pointing to always_clobber.php.
| | A release specific bug should be filed, except for b2 to bN (end of cycle). releaseduty is on the hook to pre-file these, but it should be done manually for chemspills. These bugs should: |
| | * be assigned to the releng person handling the release |
| | * ensure the relman lead is cc'd (since they start the build, and need to be aware of any infrastructure blockers) |
|
| |
|
| If you are doing a test release in staging there is [[Release:Release_Automation_on_Mercurial:Documentation#Staging_Specific_Notes|additional setup work]] to do, and the clobberer URL changes.
| | [[Releases/BuildNotesIndex|Build notes]] are required for every release, located at Releases/<app>_<version>/BuildNotes (eg [[Releases/Firefox_34.0b4/BuildNotes]], [[Releases/Firefox_33.0.2/BuildNotes]], [[Releases/Thunderbird_31.2.0/BuildNotes]]). A copy of the [[Releases/RelEngChecklist | Checklist]] should be copied in, so that work items can be marked off as they are completed. This makes handing-off releases between people more reliable. |
|
| |
|
| = Locking slaves =
| | File bugs on specific issues you hit which can be fixed by automation improvements, or for patches that require review and there is no release tracking bug. |
| Because releases happen on only one master and slaves are scattered across all of them, it's necessary to make sure you have enough slaves to complete the release in a timely manner prior to starting. Generally, you need at least 3 or 4 for each platform, but 6 is necessary to fully parallelize l10n repacks for minimum time elapsed. Use the Buildbot interface for each build and l10n platform (eg, http://buildbot-master12.build.scl1.mozilla.com:8001/builders/release-mozilla-release-win32_build and http://buildbot-master12.build.scl1.mozilla.com:8001/builders/release-mozilla-release-win32_repack_1%2F6) and verify that you have them. If you don't have enough use [http://slavealloc.build.mozilla.org/ui/#slaves Slavealloc] to lock additional slaves to your master. If you don't know how to do this, ask someone. If you the slaves you reallocate in slavealloc aren't currently connected to the master that you'd like them to be, you'll need to reboot them after you tie them to the new master.
| |
|
| |
|
| == Finding which platforms are involved in a given release ==
| | [[category:Release_Management|R]] |
| | |
| Convenient URL's for finding which platforms are involved in a given release type:
| |
| * [http://buildbot-master13.build.mozilla.org:8001/waterfall?num_events=10&category=release-mozilla-beta- FF Beta]
| |
| * [http://buildbot-master30.build.mozilla.org:8001/waterfall?category=release-comm-beta- TB Beta]
| |
| | |
| == Minimum Requirements ==
| |
| | |
| It may take a while to corral all the slaves needed. The minimum required to start a release are:
| |
| * 3 linux,
| |
| * 2 of everything else (2nd for xul-runner, so only need 1 for thunderbird)
| |
| | |
| = Setting reserved slaves =
| |
| Aside from making sure you have enough slaves attached to the master, you also need to make sure that those slaves aren't busy doing non-release jobs. For this, reserved slaves is used. It improves the odds of having slaves available when the release starts, which helps reduce end to end time. Reserved slaves is a simple file on the master that's read on a timer by the master. To set it, something like the following should be done a few hours ahead of the release starting (when possible):
| |
| cd /builds/buildbot/build1/master
| |
| echo 8 > reserved_slaves
| |
| | |
| = Post a Patch for Review =
| |
| | |
| Note: Firefox releases also include Fennec unless otherwise specified.
| |
| | |
| For a Firefox beta, you'll need to modify:
| |
| | |
| mozilla/l10n-changesets_mobile-beta.json
| |
| mozilla/l10n-changesets_mozilla-beta
| |
| mozilla/release-fennec-mozilla-beta.py
| |
| mozilla/release-firefox-mozilla-beta.py
| |
| | |
| = Starting the automation =
| |
| Generally, the workflow for kicking off Release Automation is as follows:
| |
| * Land configuration file updates. Double-land on the <code>production</code> branch if you aren't merging <code>default -> production</code>.
| |
| * Tag buildbotcustom ''(branch <tt>production-0.8</tt>)'', buildbot-configs ''(branch <tt>production</tt>)'', tools ''(branch <tt>default</tt>)'' with the correct _RELEASE and _BUILDn tags. For example, FENNEC_14_0b6_RELEASE and FENNEC_14_0b6_BUILD1. <em>Multiple production releases require tags for both products</em>.
| |
| * Update & [[ReleaseEngineering:Buildduty#Scheduled_Reconfigs|reconfig]] the master doing the release (<code>make update</code> and <code>make reconfig</code>)
| |
| * Run release sanity. For example:
| |
| # Combined release
| |
| cd /builds/buildbot/build1/master
| |
| source ../bin/activate
| |
| PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u bhearsum \
| |
| -V 6.0b2 --branch mozilla-beta --build-number 1 \
| |
| --release-config release-firefox-mozilla-beta.py \
| |
| --release-config release-fennec-mozilla-beta.py --products firefox,fennec \
| |
| --dryrun localhost:9001
| |
| # Firefox-only release
| |
| cd /builds/buildbot/build1/master
| |
| source ../bin/activate
| |
| PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u bhearsum \
| |
| -V 6.0b2 --branch mozilla-beta --build-number 1 \
| |
| --release-config release-firefox-mozilla-beta.py --products firefox \
| |
| --dryrun localhost:9001
| |
| # Thunderbird release:
| |
| cd /builds/buildbot/build1/master
| |
| source ../bin/activate
| |
| PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u hwine \
| |
| -V 14.0b5 --build-number 1 --branch comm-beta \
| |
| --release-config release-thunderbird-comm-beta.py --product thunderbird \
| |
| --dryrun localhost:9001
| |
| | |
| * Start the automation by running the sanity script again, without --dryrun.
| |
| | |
| If you're working in staging you must make sure to pass in the correct staging release config (staging_release-firefox-<branch name>.py)
| |
<< Documentation
After we get a "go to build" e-mail from Release Management there's some manual steps to be run to prepare for and start a release. This page describes them in detail.
Overall beta release
Source: File:Beta-cycle.zip
See also:
Submit to Ship It
This task is completed by Release Management
Releases are initiated through the Ship It webapp (through the Mozilla VPN). When you're ready to start a release, you can use the "Submit a new release" page to submit it. Once it's in the system you should find someone else to review it for you. They can see it and mark it as ready on the "View releases" page. If multiple releases are expected around the same time it's often best to wait until all are ready for review, so that they can be started at the same time.
Track Ship-It status on the the "View releases" page. If there are errors reported, check your email for details from "cltbld" with subject starting "[release-runner]".
Defaults
Unless otherwise stated by RelMan we stick to certain assumptions about releases:
- Firefox betas need partial updates from the previous 2 betas
- Firefox releases (that is, a release on the release channel) need partial updates from the 3 past versions with the most users. If you don't know how to get this information ask someone in RelEng or RelMan.
- A "go" for Firefox implies a "go" for Fennec, too
Paperwork
A release specific bug should be filed, except for b2 to bN (end of cycle). releaseduty is on the hook to pre-file these, but it should be done manually for chemspills. These bugs should:
- be assigned to the releng person handling the release
- ensure the relman lead is cc'd (since they start the build, and need to be aware of any infrastructure blockers)
Build notes are required for every release, located at Releases/<app>_<version>/BuildNotes (eg Releases/Firefox_34.0b4/BuildNotes, Releases/Firefox_33.0.2/BuildNotes, Releases/Thunderbird_31.2.0/BuildNotes). A copy of the Checklist should be copied in, so that work items can be marked off as they are completed. This makes handing-off releases between people more reliable.
File bugs on specific issues you hit which can be fixed by automation improvements, or for patches that require review and there is no release tracking bug.
