View source for Release Management/ESR Release Checklist Documentation

The goal of this page is to document the [https://docs.google.com/spreadsheets/d/1Aet2dJ-wW0zcBx0mS__QSWzZdAs2wKlub1oFUDHzVF4/edit?usp=sharing ESR Release Checklist] being used by Firefox Release Managers to track each release throughout the cycle. Any changes to this documentation or the checklist should be reflected in '''both''' documents.

Also see the [[Release_Management/Release_Process_Checklist_Documentation|Release Process Checklist Documentation]] for more related information.

== Ongoing Tasks ==
* '''Review pending approval-mozilla-esrXX requests.''' To avoid making life more complicated in the event of a dot release for the previous ESR release, we typically wait to start approving and landing uplifts until around the mid-point of the cycle. Doing so reduces the likelihood of needing a relbranch should a dot release need to occur (i.e. we can build off default instead). However, the approval requests should still be triaged on a regular basis to ensure that no critical issues have gone unnoticed.

== Queries ==
Sample queries:
* approval-esr60? https://mzl.la/2vrGfaB
* esr60 burndown https://mzl.la/2VxLPHv
* esr60 tracking:? https://mzl.la/2W6VSjF
* esr60.7 tracking+  https://mzl.la/2Pw8kHd

== RC Week ==
* '''Send smoketesting email to enterprise list.''' Two weeks prior to Go-Live, an email like the one shown here should be sent to the enterprise mailing list so that interested users can test the upcoming release in their environments prior to launch and report any issues when there's still time for them to be addressed. [https://mail.mozilla.org/private/enterprise/2018-November/000405.html Example]
* '''Verify all approved bugs landed on m-esrXX.''' After approving patches for uplift, they must be pushed to the appropriate ESR repository ([https://hg.mozilla.org/releases/mozilla-esr60 mozilla-esr60] at the time of this writing). This task can be performed by the [https://wiki.mozilla.org/Sheriffing Tree Sheriffs] (#sheriffs on IRC) or by the release manager themselves depending on their comfort level. The longer-term goal is to automate the process.
* '''Verify no remaining esrXX:YY+ tracked bugs.''' Confirm that all bugs tracked for this specific ESR release have been fixed.
* '''Set up builds in ship-it.''' [https://shipit.mozilla-releng.net/ Ship-it] is the tool used for scheduling the release process, starting with the creation of the builds (picking a revision, verifying the version number, etc) and the eventual pushing of those builds to the release mirrors and website. '''Access to ship-it requires being connected to the Mozilla VPN.''' Note that the ESR option should be selected instead of Beta as shown in the screenshots below.
[[Image:RelMan Process Ship-It v2 1.png|600px|center|middle]]
[[Image:RelMan Process Ship-It v2 2.png|450px|center|middle]]
* '''Treeherder tests green/starred.''' [https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta Treeherder] is the primary dashboard for monitoring the results of builds and tests. It is the responsibility of the sheriffs to monitor the Beta repository and ensure that tests are passing, though the release manager can also keep an eye on things. '''Builds should not be started until CI has passed to avoid shipping defective code to end users.'''
* '''Start build from ship-it.''' Once CI results are good, the process of generating the builds (go-to-build) is started by clicking the '''promote''' button for each release.
[[Image:RelMan Process Ship-It v2 3.png|600px|center|middle]]
* '''Confirm build has started (emails, ship-it).''' Emails will be sent to the release-signoff mailing list once builds have started and a notification will be posted in the #releaseduty IRC channel.
* '''Confirm notification sent when builds finish.''' An automated email will be sent to the release-signoff mailing lists once the release promotion process is finished.
* '''Draft release notes.''' ESR release notes are generally pretty simple (noting security and stability fixes). Feel free to add any other important bugs as well.
* '''QA manual testing signoff.''' Ask in the #qa-coordination Slack channel if there are questions about progress.
* '''Update tests on esr-localtest.''' Initial update testing performed by QA at the same time as the build sign-off.

== Go-Live Week ==
* '''Schedule push to CDN (ship-it).''' This is typically done on the day prior to launch day.
* '''Email confirmation for the push to esr-cdntest.''' Sent by automation once the CDN push has completed successfully.
* '''Update tests on esr-cdntest.''' Once QA has performed update testing on the cdntest update channel, the builds are ready to ship.
* '''Verify updates to What's New Page (if applicable).''' ESR releases rarely have What's New Pages, but can be used for new major releases.

== Launch Day ==
* '''Schedule push to esr at 100% (ship-it).''' Go-Live time is usually 6am PT on launch day, but this can be done ahead of time with a scheduled rule change in Balrog. Unlike non-ESR Firefox releases, ESR releases go out at 100% from day 1 rather than a staged rollout.
* '''Release Ubuntu's snap to the main ESR population.''' This can be done by going to https://snapcraft.io/firefox/releases and dragging the candidate version onto the stable channel on the corresponding ESR track. In the case that there are two ESR versions during the extended support period (for example 102 and 115), Canonical would need to create a new branch for the oldest version. The builds will be updated on the new branch by Canonical. When ready to ship, you must use Snapcraft CLI to release this version. Using the example of ESR102 and ESR115, ESR115 would be released to the stable channel on the esr track using the WebUI. After confirming that all builds are built on the custom branch, you would run the following command

  snapcraft promote --from-channel=esr/candidate/102 --to-channel=esr/stable/102 firefox

: If for any reason this '''cannot''' be done on the WebUI, here are some manual steps. Supposing you have docker installed on your machine, run the following commands:
  docker run -ti snapcore/snapcraft:stable
  snapcraft release firefox $SNAP_REVISION esr/stable
  snapcraft close firefox esr/candidate
: $SNAP_REVISION is a number (ex: 370) that can be found at https://dashboard.snapcraft.io/snaps/firefox/, it is not the esr release number. Closing "esr/candidate" will move the users to the "esr/stable" channel. They will get back to "esr/candidate" the day we reopen it.  
* '''Make release notes live.''' There is a 15-20min lag between making the change in Nucleus and the live website picking up this change, so plan to do this 15-30min prior to go-live.
* '''Signoff on scheduled rule change in Balrog.''' Assuming that the change is scheduled for 6am PT, this can be signed off ASAP to avoid unnecessary delays at go-live time.
* '''Verify that new release is live on mozilla.org.''' Verify that download requests are pointing to the new version from the [https://www.mozilla.org/firefox/organizations/all/ download page]. This can probably be moved to the delivery dashboard.
* '''Verify that the release notes are live.'''
* '''Verify FIREFOX_ESR in product-details.json.'''
* '''Update tests on esr.''' QA will send a sign-off email when this is completed.
* '''Security advisories go live.''' This will be handled by the security team post-launch.
* '''Send the announce email to enterprise list.''' After launch, an email like this one [https://mail.mozilla.org/pipermail/enterprise/2018-December/000429.html] is sent to the enterprise mailing list.