Confirmed users
391
edits
ReleaseEngineering/Releaseduty/FAQ: Difference between revisions
Jump to navigation
Jump to search
Polish the FAQ corresponding to the new release-promotion world.
(Add reference to relpro wiki.) |
(Polish the FAQ corresponding to the new release-promotion world.) |
||
| Line 1: | Line 1: | ||
'''0. Which communication channels should I join to know what's happening?''' | |||
Good starters are: #releaseduty | Good starters are: | ||
* #releaseduty | |||
* #tbdrivers | |||
* #release-drivers | |||
* #relman | |||
There are also mailing list subscriptions to | |||
* [https://groups.google.com/a/mozilla.com/forum/?hl=en#!forum/release-automation-notifications release-automation-notifications] | |||
* <thunderbird-drivers@mozilla.org> | |||
* <release-automation-notifications-thunderbird@mozilla.org> | |||
* <release-drivers@mozilla.org> | |||
* <release@mozilla.com> | |||
2. With respect to release-runner, I grasped the release-runner part in the graph. However the | '''1. How does the Ship-it workflow work in terms of shipping a new release?''' | ||
* | Release Manager (RelMan) submits a new release form [https://ship-it.mozilla.org/ here], another RelMan reviews that and once it hits 'Ready' + 'Do eeaat' the release enters the 'Reviewed' section and waits to be run. Since there's a `release-runner.sh` script running in a loop on `bm81`, there's a max windows of (60 seconds-ish?) till the job gets its share, following which it enters the 'Running/Complete' table where we can observe its state. The "Reviewed" tab goes to "No pending release" yet again. | ||
'''2. With respect to `release-runner.sh`, I grasped the release-runner part in the graph. However the tools/buildfarm/release/release-runner.py shows me a TC task is created.''' | |||
* Where are those operations (patch, tag, sanity check, push &reconfig, trigger) | |||
* What does the scheduler.createTaskGraph mean and the mark_as_completed part? | |||
Not long ago, the release runner tackled the releases within Buildbot world. That meant that for every release submitted through Ship-it, release runner would patch buildbot-configs, tools and buildbotcustom repos, do some release sanity check , tag and push the aforementioned repos, following which trigger a buildbot reconfig to make sure the changes are pulled down across all buildbot masters. All the scheduling part was being done in Buildbot. | |||
Everything changed when we switched to [[ReleaseEngineering/Release_build_promotion|release promotion]] during March of 2016. Long story short, that means all the task scheduling is being done within Task Cluster. Release runner uses instead Taskcluster API tools to create the graph task encompassing all underlying tasks and schedules it for running within Task Cluster. No more reconfigs, no more Buildbot! | |||
'''3. What does release-promotion refer to within all this context'''? | |||
'release promotion' is simply the idea that we take an already existing CI build from (e.g.) beta and 'promote' that to being the build we release/ship to users. Prior to this idea, we have always rebuilt Firefox at the start of each new release. The diff between the current CI builds and CI builds going forward is that they will have slightly different mozconfigs and bits that make them as ready-to-ship. The assumption is that each CI build is releasable without changing it, so the signatures are final. (yes, no more bumping versions/build configs/mozconfigs/signatures!!). Long story short, release promotion entails taking an existing set of builds that have already been created and passed QA and “promoting” them to be used as a release candidate. This represents a fundamental shift in how we deliver Firefox to end users, and as such is both very exciting and terrifying at the same time. More on promotion can be found in our wiki [[ReleaseEngineering/Release_build_promotion|here]]. | 'release promotion' is simply the idea that we take an already existing CI build from (e.g.) beta and 'promote' that to being the build we release/ship to users. Prior to this idea, we have always rebuilt Firefox at the start of each new release. The diff between the current CI builds and CI builds going forward is that they will have slightly different mozconfigs and bits that make them as ready-to-ship. The assumption is that each CI build is releasable without changing it, so the signatures are final. (yes, no more bumping versions/build configs/mozconfigs/signatures!!). Long story short, release promotion entails taking an existing set of builds that have already been created and passed QA and “promoting” them to be used as a release candidate. This represents a fundamental shift in how we deliver Firefox to end users, and as such is both very exciting and terrifying at the same time. More on promotion can be found in our wiki [[ReleaseEngineering/Release_build_promotion|here]]. | ||
Please address [[Release:Release_Automation_on_Mercurial:Preparation#Overall_beta_release|this]] schema for details. | '''4. In a few steps, how does a beta generation process looks like?''' | ||
<s>'''Please address [[Release:Release_Automation_on_Mercurial:Preparation#Overall_beta_release|this]] schema for more details. '''</s><p>The schema is longer 100% accurate as steps <4> and <5> from the schema have been wiped off once release promotion took service.</p> | |||
'''5. Somebody on IRC said "we need to update several stuff till the beta goes out in the release channel". Are all these hotfixes for stuff that has been observed since the changes started riding the trains?''' | |||
Since 2012 Mozilla moved to a fixed-schedule release model, otherwise known as the Train Model, in which we released Firefox every six weeks to get features and updates to users faster and move at the speed of the Web. Hence, every six weeks the following merges take place: | |||
[http://hg.mozilla.org/mozilla-central/ mozilla-central] => [http://hg.mozilla.org/releases/mozilla-aurora/ mozilla-aurora] | [http://hg.mozilla.org/mozilla-central/ mozilla-central] => [http://hg.mozilla.org/releases/mozilla-aurora/ mozilla-aurora] | ||
| Line 31: | Line 49: | ||
[http://hg.mozilla.org/releases/mozilla-beta/ mozilla-beta] => [http://hg.mozilla.org/releases/mozilla-release/ mozilla-release] | [http://hg.mozilla.org/releases/mozilla-beta/ mozilla-beta] => [http://hg.mozilla.org/releases/mozilla-release/ mozilla-release] | ||
So before we even ship a new release, there's a 6-8 weeks of betas in which we test things. Unexpected bugs can and usually come up in this time so there's need for hotfixes. | So before we even ship a new release, there's a 6-8 weeks of betas in which we test things. Unexpected bugs can and usually come up in this time so there's need for hotfixes. But not all the stuff is entirely hotfixes. Some of the updates are *uplifts*, hence commits that are being cherry-picked from mozilla-central to aurora/beta/release/esr*/etc. | ||
6. Overall rate vs update rate? | '''6. Overall rate vs update rate?''' | ||
* What's the overall rate everyone's keep talking about in the release channel mtg? | |||
(e.g. Overall rate: 1.0 - yellow (almost green), on the upper edge of what 42 data looked like) | (e.g. Overall rate: 1.0 - yellow (almost green), on the upper edge of what 42 data looked like) | ||
* | * when they are talking about "update rate from 30 to 100" they are talking about throttling a release? | ||
<code>TODO answer</code> | <code>TODO answer</code> | ||
'''7. Scenario: Release-runner triggers a sendchange to buildbot. Buildbot builds. Is there any automatic testing chained up as soon as the build is ready? (as in, the same testing I see on treeherder for regular granular commits?)''' | |||
''' | No. To understand that you must first understand the train model system. Any current change that's being added by developers is first tested under a specific branch in [https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound treeherder] (could be '''try''' branch for example). Assuming it has passed all the tests, the change is ready to be landed in [https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound%20treeherder mozilla-inbound] where it runs the tests again. Assuming everything is green yet-again, the Sheriffs will pull down that changset and drag it under [http://hg.mozilla.org/mozilla-central/ mozilla-central] from where it starts riding the train model. Initially riding within the Nightly, then Aurora, then Beta to finally arrive in the release channel. Whenever a hotfix/chemspill comes into place and needs to be addressed, it it tested before the code is landed in the specific mercurial code base. Therefore, it's not being actually tested within the release process but beforehand. | ||
Later edit: Once [[ReleaseEngineering/Release_build_promotion|release promotion]] shipped mid-March 2016 there's no need for re-rebuilding and re-testing as RelMan is promoting an existing CI build, which already ran the tests. | |||
'''8. Just a general overview, no need for micro-level details, how do the QA tools work'''? | |||
* their testing is subsequent to automatic testing, right? (that is regular tests from treeherder) | * their testing is subsequent to automatic testing, right? (that is regular tests from treeherder) | ||
* how do they test a build? Do they have specific tools? | * how do they test a build? Do they have specific tools? | ||
| Line 66: | Line 74: | ||
<code>TODO answer</code> | <code>TODO answer</code> | ||
'''9. What is a partner repack change for FF? What does "partner" refer to in this context? (e.g. {{bug|1231679}})''' | |||
Partner repacks refer to 3rd party customized branded versions of Firefox that Mozilla is taking care of for some of its clients. With some exceptions, most of the partner reconfigs lie under private repositories. For example, whenever we ship a build release on our servers, we also take care and host a Bing repack. For the others that we don't publish, all the resulting artifacts are private. Mostly, the partner repacks don't need too much of RelEng interference as all bits are held under private git repos and are directly handled by the partnering companies. <s>'''The usual first occurrence with them for a releaseduty is when releases may get some delays in being published through the ''<channel>-cdntest'' because some of the partner-repacks are still running, hence holding the checksums step which at its turn holds the push-to-mirrors step.''' </s> Once the release promotion has shipped, partner repacks are running separately and block no other task within the release process. They depend solely on the push-to-candidates flag and the l10n platforms they're ought to be run within. | |||
'''10. Is there a programmed calendar for the Thunderbird events? As it is for FF desktop or it's just something that is being setup in the #release-drivers and synced-up with TB folks?''' | |||
We don't have a fixed calendar for Thunderbird. Developers are taking care of it internally and notify us whenever they are ready to ship. The converations are happening in #tbdrivers on IRC but the request to push "officially" comes by email to release-drivers@ email address. | |||
'''11. I got confused by the order of the things happening.''' | |||
The below answer relates to the pre-release-promotion era, in which we used to rebuild and test a release, once decided upon its hash target. Ever since we switched to release promotion, the build already exists and it's being tested. Among the remaining tasks to complete are l10n repacks, partial computation, balrog submission, signing, etc. But the main en_US binaries are already there. | |||
Even though the context below is already outdated, it's useful to understand the past logic and how release promotion simplified the things. | |||
<code> | |||
------ | |||
As soon as the build is completed and available, the QE starts testing it? (manual, regresion, smoking, etc) | As soon as the build is completed and available, the QE starts testing it? (manual, regresion, smoking, etc) | ||
I ask that because earlier today I was debugging some firefox_antivirus and update_verify_release on win64 steps when I saw an email like "[desktop] Firefox 43.0.4 (build 3) - Sign Off on Manual Functional Testing". | I ask that because earlier today I was debugging some firefox_antivirus and update_verify_release on win64 steps when I saw an email like "[desktop] Firefox 43.0.4 (build 3) - Sign Off on Manual Functional Testing". | ||
| Line 110: | Line 121: | ||
* ''[release] Firefox 45.0b2 build1: Updates available on beta'' | * ''[release] Firefox 45.0b2 build1: Updates available on beta'' | ||
* ''[release] Firefox 45.0b2 build1: completed firefox_postrelease'' | * ''[release] Firefox 45.0b2 build1: completed firefox_postrelease'' | ||
------ | |||
</code> | |||
'''12. How come Windows64 is not in the platform list for 38.6.0esr?''' | |||
We only added support for Windows64 just recently, during the fall of 2015. | |||
'''13. Email stages I've noticed for <release> and <beta>. Are these stages correct?''' | |||
'' | The below answer relates to the pre-release-promotion era. Even though the context below is already outdated, it's useful to understand the past logic and how release promotion simplified the things. At this point we no longer have emails to follow the logic real-time, but soon, once {{bug|1253369}} ships we'll get back on track with those. | ||
<code> | |||
------ | |||
'''release''' | '''release''' | ||
| Line 143: | Line 156: | ||
* release-automation sends notification stating that: "[release] Firefox 43.0.3 build1: completed firefox_postrelease" | * release-automation sends notification stating that: "[release] Firefox 43.0.3 build1: completed firefox_postrelease" | ||
* done | * done | ||
'''betas''' | '''betas''' | ||
| Line 161: | Line 172: | ||
* done | * done | ||
------ | |||
</code> | |||
Questions: | Questions: | ||
* My explanation so far is that betas are auto when it comes to push-to-mirrors, | * My explanation so far is that betas are auto when it comes to push-to-mirrors, whilst for any other releases this step is to be done manually. '''Is this correct?''' | ||
> Yes! | > Yes! | ||
* '''what does the "updates testing" refer to?''' I saw it performed at various steps during the release process. Is it automated / stage? | * '''what does the "updates testing" refer to?''' I saw it performed at various steps during the release process. Is it automated / stage? | ||
| Line 171: | Line 183: | ||
> All of the releases have the <channel>-localtest, <channel>-cdntest and <channel>. All of them require signing-off QE for each of these and update testing from QE/release-drivers. Once this is done, releng moves it forward, either manually or automaically (e.g. from <channel>-localtest to <channel>-cdntest for beta releases). | > All of the releases have the <channel>-localtest, <channel>-cdntest and <channel>. All of them require signing-off QE for each of these and update testing from QE/release-drivers. Once this is done, releng moves it forward, either manually or automaically (e.g. from <channel>-localtest to <channel>-cdntest for beta releases). | ||
'''14. Why don't I see update_verify_beta for dot releases?''' | |||
From time to time, a handful of issues precipitate a dot release. When that happens, its behavior slightly varies from a normal release. A normal release (e.g. 43.0, 44.0, etc) has its RC shipped to beta channel first before making it to release channel - for testing purposes, update verify steps are taking place both ways, hence update_verify_release '''and''' update_verify_beta steps. Upon successful testing we ship the RC on the beta channel and then on the release channel, following which we merge the code for the next release cycle so that the beta release bumps its version. In the lights of this logic, a dot release (e.g. 43.0.1 or 44.0.1) happens a certain amount of time '''after''' the official release. For that reason, a dot release can't be tested in beta channel as the at-that-moment beta version is greater than the dot release version, hence the updater would refuse to downgrade. Therefore, there is only one cycle of update_verify for dot releases (update_verify_release == update_verify in this case). | From time to time, a handful of issues precipitate a dot release. When that happens, its behavior slightly varies from a normal release. A normal release (e.g. 43.0, 44.0, etc) has its RC shipped to beta channel first before making it to release channel - for testing purposes, update verify steps are taking place both ways, hence update_verify_release '''and''' update_verify_beta steps. Upon successful testing we ship the RC on the beta channel and then on the release channel, following which we merge the code for the next release cycle so that the beta release bumps its version. In the lights of this logic, a dot release (e.g. 43.0.1 or 44.0.1) happens a certain amount of time '''after''' the official release. For that reason, a dot release can't be tested in beta channel as the at-that-moment beta version is greater than the dot release version, hence the updater would refuse to downgrade. Therefore, there is only one cycle of update_verify for dot releases (update_verify_release == update_verify in this case). | ||
'''15. As part of the release process, which step takes the user's Firefox to prompt the "There is a new update. Please click below [...]"?''' | |||
The tl;dr is "Publish to Balrog". That is the magic step that makes the hot stuff available to all the users. However, there are certain bits to complete the equitation here. Once a release completes its deliverables and its signing is done - that release is available in the release-localtest channel. After the release passes by the push-to-mirrors step it arrives in the release-cdntest channel. Finally, when the release is being pushed to balrog, it reaches to its release channel. Normally, each of these steps require a QE sign-off (and, for some, a RelMan approval). | The tl;dr is "Publish to Balrog". That is the magic step that makes the hot stuff available to all the users. However, there are certain bits to complete the equitation here. Once a release completes its deliverables and its signing is done - that release is available in the release-localtest channel. After the release passes by the push-to-mirrors step it arrives in the release-cdntest channel. Finally, when the release is being pushed to balrog, it reaches to its release channel. Normally, each of these steps require a QE sign-off (and, for some, a RelMan approval). | ||
| Line 182: | Line 196: | ||
Worth to mention here that there is a Ship-it option that enforces update with immediate or planned action. The option is controlled by RelMan whenever they trigger the release in Ship-it. | Worth to mention here that there is a Ship-it option that enforces update with immediate or planned action. The option is controlled by RelMan whenever they trigger the release in Ship-it. | ||
'''16. What are the tools that I should be aware of whilst releaseduty-ing?''' | |||
There are some existing tools that aim to ease the job of releaseduty and make it as automated as possible. These are: | |||
* [https://github.com/lundjordan/releasewarrior releasewarrior] | |||
* [https://github.com/mozilla/tctalker tctalker] | |||
<code>''' More info on these soon '''</code> | |||
== Misc info == | == Misc info == | ||
* we don't usually publish updates for the first release of esr, usually there are 2 cycles of overlap (e.g. 45.2.0 will be probably the first release when we can offer updates to 38 users) | * we don't usually publish updates for the first release of esr, usually there are 2 cycles of overlap (e.g. 45.2.0 will be probably the first release when we can offer updates to 38 users) | ||