Release Management/Release Process Checklist Documentation

The goal of this page is to document the Release Process Checklist being used by Firefox Release Managers to track each release throughout the cycle. Any changes to this documentation or the checklist should be reflected in both documents.

Nightly Checklist

Given the nature of how nightly builds are created and shipped, the role of the release manager during this phase of the cycle skews much more heavily to the monitoring aspect rather than release mechanics.

Prior to the start of the Nightly cycle, the following tasks need to be performed before Merge Day

1. Create a Release Checklist before the cycle starts.
   • Use the template from Google Drive.
2. Create the Milestones Document for the Release Checklist.
   • Use the sheet linked in the Release Checklist.
3. Create a Firefox Retrospective Notes document.
   • Use the template from Google drive.
   • It is useful to document during the release cycle.
4. Create a folder in the Release Tracking Google Drive for the release.
   • Move the Release Checklist and Firefox Retrospective Notes to this folder.
5. Create a release notes skeleton in Nucleus.
   1. Click Admin Interface
   2. Click the Releases.
   3. Select the previous Nightly release notes (enable the checkbox beside the release).
   4. At the top of the page, expand the Action dropdown
   5. Select Copy releases.
   6. Click Go.
   7. Select the copy.
   8. In the Version text box edit the version to the current Nightly version.
      • Example: 99.0a1
   9. Set the Release Date to match when Central was bumped to Nightly
   10. Remove the release notes from the previous release.
   11. For additional information on the Release Notes process see The Firefox release notes process wiki page.
6. Ensure that Bugzilla is bumped to the Nightly version, see BMO/new-version
   • Note: This is performed on the Friday before Merge Day.
   • Find the bug that tracks the bump.
     • Example bug
     • Please Note: The Milestone bump is performed by Bugzilla admins, see the ticket assignee.
   1. Update the Release Tracking flags in Bugzilla
   2. Go to Bugzilla > Administration > Release Tracking Flags
   3. Add the cf_tracking_firefox and cf_status_firefox for the Nightly release
      • Example: If 97 is becoming the Nightly, then add cf_tracking_firefox97 and cf_status_firefox97.
   4. Deactivate the oldest cf_tracking_firefox flag.
      • Example: If 95 is going to Release, 96 is going to Beta, and 97 is becoming the Nightly, then deactivate cf_status_firefox94.
   5. Edit the current cf_tracking_firefox_esr tracking flag to add the new release as a Value and deactivate the oldest release.
      • Example: If esr is based on 91, 95 is going to Release, 96 is going to Beta, and 97 is becoming the Nightly, then edit cf_tracking_firefox_esr91 to add a Value 97+ set to mozilla-next-drivers and deactivate the Value 94+
   6. Add the cf_tracking_thunderbird and cf_tracking_thunderbird for the Nightly release.
      • Example: If 97 is becoming the Nightly, then add cf_tracking_thunderbird97 and cf_tracking_thunderbird97
   7. Deactivate the oldest cf_tracking_thunderbird flag.
      • Example: If 95 is going to Release, 96 is going to Beta, and 97 is becoming the Nightly, then deactivate cf_tracking_thunderbird94.
   8. Edit the current cf_tracking_thunderbird_esr tracking flag to add the new release as a Value and deactivate the oldest release.
      • Example: If esr is based on 91, 95 is going to Release, 96 is going to Beta, and 97 is becoming the Nightly, then edit cf_tracking_firefox_esr91 to add a Value 97+ set to mozilla-next-drivers and deactivate the Value 94+.
   9. Add a comment on the bug that the tracking flags were updated.
      • See comment here for example.
7. Update the Google calendar Releases Scheduling.
   1. Open the following URL and change the version to the Nightly release.
      • https://whattrainisitnow.com/calendar/release/schedule/?version=118.
   2. Download the generated ical file.
   3. Review locally on a calendar app of your choice
   4. Open Google Calendar.
   5. Click + beside Other Calendars.
   6. Select Import.
   7. Select the ical file previously downloaded.
   8. Select the Releases Scheduling calendar.
   9. Click Import.
   10. Please Note: Importing the ical to Google Calendar is a one-shot operation. If you notice incorrect info or dates after importing, correct them directly in Google Calendar. Importing the ical file again will cause duplication.
8. Ensure the REO (Regression Engineering Owner) is listed on Release Owners.
   • If required check the Google Sheet REO Rotation by Director.
   • Update the wiki with information from the spreadsheet.
     • Ensure that the REO and RelEng owners are listed.
     • Ensure the dates are accurate.
   • If required reach out to the Director to find out who the REO is.
   • Ensure they are invited to the Weekly Regression Triage.
9. Ensure the Release Engineer on Release Duty is listed on Release Owners.
   • If required reach out on Matrix #releaseduty.

At the start of the Nightly cycle, the following tasks need to be performed on Merge Day

1. Verify that Central was bumped to the Nightly version.
   • Check version_display.txt
   • Please Note: This is performed on Merge Day of the release going to beta, if it is incorrect talk to the RelMan on duty for the Beta cycle.
2. Verify that the firefox-android version was updated.
   • Chheck version.txt should be xxx.0a1, where xxx matches the nightly version of desktop.
   • If the version was not updated, then sync with release engineering.
3. Make the release notes public in Nucleus
4. Verify the application services version was updated
   • The version in version.txt should be xxx.0a1, where xxx matches the nightly version of desktop.
   • If the version was not updated then talk to the RelMan on duty for the Beta cycle.
5. Verify the firefox_versions.json is correct.
   • Please Note: Usually this json update is performed the day after merge day
   • Please Note: fx trains and various scripts in other tools use this data.
   • Verify the following are correct:
     • FIREFOX_NIGHTLY, NEXT_MERGE_DATE, NEXT_SOFTFREEZE_DATE
   • If these are incorrect, contact the RelEng on duty via the #release-duty channel in Matrix. It is possible a PR was missed during the RelEng merge day steps.
   • Please Note: There are also date changes that are monitored by autonag. For example, if Template:NextReleaseDate was not updated an autonag email will be sent.
6. Verify the mobile_versions.json is correct
   • Verify the following are correct:
     • alpha_version, nightly_version
   • If these are incorrect, contact the RelEng on duty via the #release-duty channel in Matrix.

On a daily (or thereabouts) basis, the following items should be monitored during the Nightly cycle

1. Review Pending tracking-firefoxXX requests
   • Example Filter
   • Ensure that the tickets are progressing, follow up as required.
2. Review Open tracking-firefoxXX+ and blocking bugs
   • Example Filter
   • Ensure that the tickets are progressing, follow up as required.
3. Review incoming regression bugs
   • Example Filter
   • Also see the REO section in the BugDash
   • Ensure that new tickets are prioritized.
   • Ensure that carry-over tickets are still on the team’s radar.
   • Review with the REO in the weekly regression meeting.
4. Review stability rates and reported crash spikes
   • Leverage information from the automated emails, monitor top crashes in Crash Stats, and monitor Stability & Release Monitoring Looker Dashboard
   • Ensure that there are bugs tracking stability rates and crash spikes.
   • Ensure these tickets are tracked and prioritized.
   • Please Note: If ever there are stats/bugs that cannot be explained/identified, then reach out to the team for assistance on the #stability Matrix channel, it may be an issue with the tooling issue.
5. Review the Release Notes requests for Nightly Tickets.
   • See Release_Notes#Daily_during_the_Nightly_cycle for details.

On a weekly (or thereabouts) basis, the following items should be monitored during the Nightly cycle

1. Review the tracking Mana page for the release.
   • The tracking page can be found as a child of Firefox Release Tracking
   • Ensure that features targeting the release are on track, reach out as needed if any are slipping.
2. Update the Weekly Release Tracking Digest document, as requested on #fx-weekly-digest.
   • Update the Release Milestones (Details) section to add Status (On Track, Delayed), Ship Date, and additional details depending on where the nightly is in the cycle for example coming up to the Soft Freeze.
   • Review the dates in the Release Milestones section and update the status.

As available, the following items should be monitored during the Nightly cycle

1. Review and land mozilla-central repo-update HSTS HPKP remote-settings tld-suffixes patches.
   • Every Monday and Thursday revisions are automatically created for review in Phabricator.
   • Monitor email notifications sent by Phabricator: “No Bug, mozilla-central repo-update HSTS HPKP remote-settings tld-suffixes”
   • Review and land the patch to autoland.
2. Review and land android nightly application-services version update bumps to mozilla-central
   • An automated nightly application-services build is triggered daily. The build will only run if commits merged to main since the previous nightly build.
   • update-bot creates a patch to bump the version for android in mozilla-central, runs a Try build, and sets the release-mangers group as reviewer.
   • Monitor email notifications sent by Phabricator: Update android nightly application-services version bump
   • Review and land the patch to autoland.
   • If the Try build failed:
     • Check the failure.
     • Check the possible commit in application-services main that corresponds to the failure.
     • Start a thread on #application-services-eng for investigation.
3. Review Test Plans for features targeting the release. QA shares test plans via email. For example:
   • Become familiar with how the feature works.
   • Understand how the team ensures it is of sufficient quality to ship in that release.
   • Provide feedback on the risk analysis and mitigations put in place.
4. Review mid-Nightly and pre-Beta testing reports QA shares mid-Nightly and pre-Beta testing reports via email. For example:
   • If there are any red or yellow reports review the recommendations/plan and the bugs mentioned in the report.
   • If QA flagged that the feature is behind a pref verify that this is expected.

At the end of the Nightly cycle, the following tasks need to be performed prior to and during Merge Day

1. Send Nightly Soft Code Freeze reminder to dev-platform and firefox-dev.
   • Send this at the start of the week before the Soft Code Freeze.
   • Remind developers that the window for landing riskier fixes is coming to a close until after the version bump.
   • Example email
2. Sync with RelEng to confirm the RelEng on duty for the Central to Beta merge.
   • Sync on Friday before Merge Day
   • Use the #releaseduty channel on Matrix.
   • Align to perform the merge early on Merge Day, for example, 9/10am ET.
3. Sync with the Sheriff team to confirm Central is in a good state so that the Merge Day email can be sent.
   • Sync on Merge Day morning
   • Use the #sheriffs channel on Matrix.
   • Check the Beta simulations document for the release, this document is emailed to Release Management at the start of the night cycle.
   • Ensure that central does not currently have any known severe regressions or performance issues that would be carried into beta.
4. Once you have confirmed central is in a good state, send the Merge Day email to release-drivers and release signoff
   • Example email for the merge day of 103:
5. Sync with RelEng on the Central to Beta merge
   • Use the #releaseduty channel on Matrix to mention the merge day email was sent
   • Sync with the RelEng on duty. The RelEng on duty is listed in the Matrix channel description
6. Create a Desktop Beta Checklist and Mobile Beta Checklist for beta 1
   • Use the b1 macro
7. Wait for RelEng to complete the central to beta merge.
   • Use the #releaseduty channel on Matrix to synchronize if needed.
   • RelEng will also reply to the Merge Day email
8. Prepare the Beta release notes in Nucleus
   1. Click Admin Interface
   2. Click the Releases
   3. Select the Nightly release notes (enable the checkbox beside the release)
   4. At the top of the page, expand the Action dropdown
   5. Select Copy release
   6. Click Go
   7. In the Channel dropdown select Beta
   8. In the Version text box edit the version to the Beta version
      • Example: 96.0beta
   9. Replace the content in the Text box with content from a previous beta release
   10. Set the Release Date to the date when the first Beta is released
   11. Ensure to remove any notes that are only applicable to nightly
9. Announce the soft freeze is over
   • Once the merge is complete, reply to the soft freeze reminder email
   • Example: “Hi, the Gecko 100 version bump landed on central as planned and the soft freeze is over.”

Beta Checklist

Once a release moves to the Beta cycle, the daily tasks performed during the Nightly cycle will continue to be carried out as new bug reports come in from a wider audience and new features move through the QA cycle towards shipping. There is also an added triage step during Beta - monitoring the “missed uplifts” email or queries to find issues fixed in Nightly that still affect Beta.

The following tasks need to be performed on Merge Day at the start of the Beta cycle after the merge to beta is finished

1. Perform the release management steps in the App Services Release checklist
   • Application Services release once and at the start of the beta cycle.
   • This step is required before building Android/iOS.
2. Switch to Application Services in Firefox Android to Release
   • Change the Version and Channel in mobile/android/android-components/plugins/dependencies/src/main/java/ApplicationServices.kt
     • See example commit
   • Push directly mozilla-beta
3. Perform the release management steps in the Firefox iOS Soft Freeze Checklist
   • The Firefox iOS release branch is used for beta and rc builds.
4. Announce the soft freeze is over
   • Reply back to the soft code freeze announcement email
   • "The version bump to xxx has now landed on mozilla-central and the soft freeze period is over."
5. Review any pending uplifts that may be required before proceeding with a beta 1 build
   • This should not be a common occurrence, however, an example would be a crash fix that may have landed in central post-merge.
6. Set up Desktop and DevEdition builds in Ship-It
   1. Connect to the VPN
   2. Log into Ship-It
   3. Click Releases dropdown
   4. Click Firefox dropdown
   5. Click New
      • 
   6. For Desktop complete as follows and click Create Release:
      • Please note: Revision should be the tip. The Partial updates field is automatic.
      • 
   7. For DevEdition complete as follows and click Create Release
      • Please note: Revision should be the tip. The Partial updates field is automatic.
      • 
7. Start the Desktop and DevEdition in Ship-it via the Promote Task
   1. View the Pending Releases in Ship-It
   2. Click the Promote Task for Desktop and continue through the pop-ups
   3. Click the Promote Task for DevEditon and continue through the pop-ups
8. Confirm the Desktop and DevEdition builds have started
   • Taskcluster email: “[desktop] Build of firefox xxx.0b1 build 1”
   • Taskcluster email: “[desktop] Build of devedition xxx.0b1 build 1”
9. Confirm notification sent when the Desktop and DevEdition builds finish
   • Taskcluser email: “firefox xxx.0b1 build1/mozilla-beta is in the candidates directory”
   • Taskcluser email: “devedition xxx.0b1 build1/mozilla-beta is in the candidates directory”
10. Schedule push to Desktop and DevEditon CDN from Ship-It via Push
    1. Click Push
       • 
    2. Click Schedule
       • 
11. Confirm notification sent when the Desktop and DevEditon CDN push finishes
    • Taskcluser email: “firefox xxx.0b1 build1/mozilla-beta has been pushed to cdntest”
    • Taskcluser email: “devedition xxx.0b1 build1/mozilla-beta has been pushed to cdntest”
12. Create Firefox Android (AC, Fenix, Focus) release in Ship-It
13. Start Firefox Android (AC, Fenix, Focus) release in Ship-It via Promote
    • Please note: you can only start a ship-it build on a revision that has a completed decision task
14. Confirm notification sent when the Firefox Android (AC, Fenix, Focus) builds finish
    • Taskcluster email: “firefox-android XXX.0b# build1/mozilla-beta is in the candidates directory"
15. Push Firefox Android (AC, Fenix, Focus) release in Ship-It via Push
16. Monitor for QA sign-off on desktop functional testing, before proceeding with Step 17.
    • Please Note: Desktop Build Validation sign-off is usually provided the day after Merge Day.
    • QA will post a message to the #qa-coordination channel in Slack when they complete functional testing.
    • Build validation testing is tracked here
17. Once QA has signed off, push Desktop and DevEdition to Beta in Ship-It via Ship.
18. Verify that the Balrog rule changes are live and correctly set.
    • View the Balrog Rules for Firefox Beta
      • Verify that the Background rate for Firefox-xxx.0b1-build1 is 25
    • View the Balrog Rules for Aurora (DevEdition)
      • Verify that the Background rate for Devedition-xxx.0b1-build1 is 100
19. Activate automated betas in Ship-It
    1. At the bottom of the page, click the red x beside Firefox Desktop: Beta
       • 
    2. Click Enable
       • 
    3. Perform the same actions for Firefox Developer Edition: Beta
    • Please Note: Automated Betas are built at 21:00 UTC every Sunday, Tuesday, and Thursday as determined here.
20. Make the Beta release notes public in Nucleus
    • Enable the Is Public checkbox and save
      • 
21. Share a link to the Beta release notes and the release notes draft document in #release-notes Slack channel
    • See example
22. Email release-notes with a link to the draft document and the submission deadline.
    • See example
23. Monitor for QA sign-off on desktop update testing
    • QA will post a message to the #qa-coordination channel in Slack when they complete update testing.
24. Monitor for QA sign-off on Fenix/Focus build validation
    • Fenix build validation testing is tracked here
    • Focus build validation testing is tracked here
    • Please Note: Build Validation sign-off is usually provided the day after Fenix/Focus builds are produced.
    • Focus and Fenix QA sign-off are sent via #qa-coordination Slack Channel
    • Once QA has signed off, roll out Fenix to the production track at 25% in Google Play
      • See Firefox for Android Beta
    • Once QA has signed off, roll out Focus to the Closed testing - Foxfooding track at 25% in Google Play
      • See Firefox Focus (Beta for Testers)
25. Monitor for QA sign-off on Firefox iOS build validation
    • Build validation testing is tracked here
    • Once QA has signed off, add the External Beta Testers group to the Firefox iOS TestFlight Build
26. Monitor for QA sign-off on Focus iOS build validation
    • Build validation testing is tracked here
27. Verify the mobile_versions.json is correct
    • Verify the following beta_version is correct:
    • If this is incorrect, contact the RelEng on duty via the #release-duty channel in Matrix.

The following tasks need to be performed daily during the Beta cycle

1. Monitor Uplift Requests
   • Example Filter and Phabricator Dashboard
   • Accept or reject the beta uplift requests.
   • See Uplift Rules for guidelines
   • Please Note: During the RC week, also monitor release uplift requests.
   • Please Note: When pushing uplifts be conscious of timing around the automated beta builds. Pushing an uplift will trigger CI, the automated beta build will not trigger until the CI completes.
   • Please Note: If an uplift contains string changes, then l10n approval is required and the commit message must end with l10n=approver_name. Add a ni on the bug for approval before taking the uplift.
2. Monitor Pending tracking-firefoxXX requests
   • Example Filter
   • Ensure that the tickets are progressing, and follow up as required.
3. Monitor Open tracking-firefoxXX+ and blocking bugs
   • Example Filter
   • Ensure that the tickets are progressing, and follow up as required.
4. Monitor Newly-filed regression bugs
   • New regressions bugs can be found here
   • Review with the REO - ensure that new tickets are prioritized and ensure that carry-over tickets are still on the team’s radar
5. Monitor stability rates and reported crash spikes
   • Leverage the automated emails, monitor top crashes in crash-stats, and monitor Stability & Release Monitoring
   • Ensure that there are tickets tracking issues with stability rates and crash spikes
6. Monitor the Release Notes requests for Beta Tickets.
   • Example Filter
   • Check through tickets in the release that are tagged for release notes. Review and add these notes to the Beta Release Notes in Nucleus.

The following tasks need to be performed for each beta build (Monday, Wednesday, Friday) during the Beta cycle

Please Note: The Beta build pipeline is also monitored by the Sheriffs, if there is an intermittent test failure they will rerun the failed job. If there is a permanent failure or an issue in the build infrastructure, then it may require canceling the build. Discuss in the #releaseduty Matrix channel. If the decision is made to cancel, then view the pending release in ship-it and click cancel. To create a new build, follow the same steps as outlined in One Time - Start of Beta Cycle to produce build 2 of the current beta release.

1. Create a tab in the release tracking sheet “xxx.0bx”
   • Use the relevant beta macro
   • Please note: After all the beta steps are complete, the tab can be hidden in order to minimize clutter.
2. Review tracking-firefoxXX+ bugs and approval requests.
   • This step is performed as part of the daily activities, tracked in the checklist for confirmation.
3. Verify all approved bugs landed on mozilla-beta.
   • This step is performed as part of the daily activities, tracked in the checklist for confirmation.
4. Ensure the Firefox iOS build is available in Testflight
   • Firefox Beta TestFlight
   • If the build is not pushed then check the build status in the BitRise pipeline
5. Confirm the Desktop and DevEdition builds have started.
   • Taskcluster email: “[desktop] Build of firefox xx.0bx build 1”
   • Taskcluster email: “[desktop] Build of devedition xx.0bx build 1”
   • Please Note: The automated beta builds schedule is determined here.
6. Confirm notification sent when the Desktop and DevEdition builds finish.
   • Taskcluster email: “firefox xx.0bx build1/mozilla-beta is in the candidates directory”
   • Taskcluster email: “devedition xx.0bx build1/mozilla-beta is in the candidates directory”
7. Create Firefox Android (AC, Fenix, Focus) release in Ship-It
8. Start Firefox Android (AC, Fenix, Focus) release in Ship-It via Promote
9. Confirm notification sent when the Firefox Android (AC, Fenix, Focus) builds finish
   • Taskcluster email: “firefox-android xx.0bx build1/mozilla-beta is in the candidates directory"
10. Push Firefox Android (AC, Fenix, Focus) release in Ship-It via Push
11. Confirm notification sent when the Firefox Android (AC, Fenix, Focus) push finishes
    • Focus/Fenix XXXb#-build1 has been pushed to the closed testing track on Google Play
12. Ship Firefox Android (AC, Fenix, Focus) from Ship-It via Ship
13. Promote Fenix to the Production track @ 50% b2 and @ 99% b3+
    • See Firefox for Android Beta
14. Promote Android Focus to the closed testing track (Foxfooding) @ 50% b2 and @ 99% b3+
    • See Focus (Beta for Testers)
15. Confirm notification sent when the Desktop and DevEditon CDN push finishes.
    • Taskcluster email: “firefox xx.0bx build1/mozilla-beta has been pushed to cdntest”
    • Taskcluster email: “devedition xx.0bx build1/mozilla-beta has been pushed to cdntest”
16. Verify that the Desktop and DevEdition Balrog rule changes are live and correctly set.
    • View the Balrog Rules for Firefox Beta
      • Verify that the Background rate for Firefox-xx.0bx-build1 is:
      • 50 for Beta 2
      • 100 for Beta 3 and greater
    • View the Balrog Rules for Aurora (DevEdition)
      • Verify that the Background rate for Devedition-xx.0b1-buildx is 100.
17. Monitor for QA sign-off on Desktop and DevEdition build validation for both update/functional testing.
    • QA will post a message to the #qa-coordination channel in Slack when they complete update testing and another message when they complete functional testing.
    • Build validation testing is tracked here
18. Monitor for QA sign-off on Fenix/Focus build validation before proceeding with bumping the Fenix/Focus rollout.
    • Focus and Fenix QA sign-off are sent via #qa-coordination Slack Channel.
    • Please Note: Build Validation sign-off is usually provided the day after Fenix/Focus builds are produced. If the sign-off is yellow/red, then follow-up on the #mobile-android-team Slack channel. Ensure a plan is in place or a decision is made to block/proceed with the release.
    • For b3+ once QA has signed off, roll out Fenix to the production track in Google Play to 100%.
      • Reply to the QA sign-off Slack message and include the rollout percentage.
    • For b3+, once QA has signed off, roll out Focus to the Closed testing - Foxfooding track in Google Play to 100%.
      • Reply to the QA sign-off Slack message and include the rollout percentage.
19. Monitor for QA sign-off on Firefox iOS build validation before proceeding with adding the External Testers group in TestFlight.
    • Firefox iOS QA sign-off are sent via #qa-coordination Slack Channel.
    • Please Note: Build Validation sign-off is usually provided the day after Firefox iOS builds are produced. If the sign-off is yellow/red, then follow-up on the #firefox-ios-releases Slack channel. Ensure a plan is in place or a decision is made to block/proceed with the release.
    • Once QA has signed off add the External Testers group to Firefox iOS.
      • Reply to the QA sign-off Slack message.

As available, the following items should be monitored during the Beta cycle

1. QA shares beta testing and pre-release testing reports via email. Review the test reports for the release. For example:
   • If there are any red or yellow reports, review the recommendations/plan and the bugs mentioned in the report. For example:
     • Track the bug reported as blockers.
     • If there is no clear plan to address the report, follow up to ensure that engineering is reviewing.
   • If QA mentions the feature is behind a pref verify that this is expected.
2. Phabricator patches for remote-settings updates are generated automatically. Typically this is every Monday and Thursday. These patches must be included in the beta uplifts.
   • Monitor email notifications sent by Phabricator: “No Bug, mozilla-beta repo-update HSTS HPKP remote-settings tld-suffixes”
   • Review the patch.
   • Commit the patch to beta via the moz-phab tool
     • moz-phab patch <patch id> --no-bookmark --apply-to here
   • Edit the commit message to add the approval.
3. Use the Mana Beta Quality Metrics Page to review the Beta Quality Metrics.
   • The data is embedded from this wiki page.
   • The data is currently added manually.
     • The data is stored automatically every day in bugzilla using New Charts
     • To extract the data, run the report by adding the next needed dates and then exporting to csv. (The queries are numbered to make this step easier)
     • This is then manually pasted into the 'Data from Queries' tab.
     • Please Note: the goal is to automate this.
   • Take a look at the metric/charts over time to ensure the value is stable.
     • Please Note: Ensure the data is updated prior to the Tuesday channel meeting.
4. Update the Weekly Digest document, as requested on the #fx-weekly-digest Slack channel.
   • Update the Release Milestones (Details) section to add Status (On Track, Delayed), Ship Date, information on the current beta release for both Desktop/DeveEdition and Fenix/Focus, information on where we are in the Beta cycle for example when coming up to end of early beta.
   • Review the Release Milestones section dates and update the status.
   • Update the Quality Metrics section to add information on the number of outstanding regressions, if there are any release blockers, and an overview on the trend of the Beta Quality Metrics metrics.

The following needs to be performed at the End of Early Beta

Please Note: This is performed on the Friday at the midday point of the beta cycle, see the Releases Scheduling google calendar. This is typically after beta 6 has been released. This should be done in its own push and before pushing any other uplifts to beta.

1. Open build/defines.sh
2. Clear the EARLY_BETA_OR_EARLIER flag:
   • EARLY_BETA_OR_EARLIER=1 to EARLY_BETA_OR_EARLIER=
3. Commit the changes:
   • hg commit -m "No bug - post beta 6, unset EARLY_BETA_OR_EARLIER. a=me"
4. Push the changes to beta.

The following needs to be performed at the end of the Beta Cycle

1. Disable automated beta builds in Ship-it.
2. Disable the firefox-ios firefox-ios SPM_Deploy_Prod_Beta BitRise scheduled workflow.
   • Expand the Scheduled section in the firefox-ios on BitRise
   • Pause the schedule for the SPM_Deploy_Prod_Beta that runs "Every Tue, Thu, Sun @ 20:00"
3. Please Note: Monitor beta/release uplift requests between the final beta build and the beta to release merge. It is possible that a late uplift request may come in for a regression/stability fix, evaluate if this is a blocker for the RC and if it should be uplifted to beta before the beta to release merge and RC build.

RC Checklist

Once a release moves to the Release cycle, preparations for the release candidates begin. Until go-live, the daily tasks are similar to the beta cycle in terms of monitoring and if needed create a new release candidate.

The following tasks need to be performed at the start of RC week (week before go-live) for Desktop/Android

1. Review tracking-firefoxXXX+ bugs and approval requests
   • Verify that all approved uplift requests were uplifted to beta
2. Verify all approved bugs landed on mozilla-release
   • Sync with RelEng to merge Beta to Release.
     • Send an email with details of the merge.
     • Include both release-drivers and release sign-off
     • Example email
3. Once the merge is complete, verify that the Treeherder tests green/starred
4. Set up Desktop build in Ship-It.
   1. Connect to the VPN.
   2. Log into Ship-It.
   3. Click New Release.
      • 
   4. Complete as follows and click Create Release:
      • 
      • Please note: Revision should be the tip. The Partial updates field is automatic.
   5. Click Submit.
      • 
5. Start the Desktop build from Ship-It via Promote RC.
   • 
   • Click the Schedule button.
     • 
6. Confirm Desktop build has started.
   • Taskcluster email: “[desktop] Build of firefox xx.0 build 1”
7. Set up Firefox Android (AC, Fenix, Focus) release in Ship-It.
   • Follow the same steps as creating a beta build.
8. Start Firefox Android (AC, Fenix, Focus) release in Ship-It via Promote.
9. Confirm notification sent when Firefox Android builds finish.
10. Push Firefox Android (AC, Fenix, Focus) release in Ship-It via Push.
11. Confirm notification sent when the Firefox Android (AC, Fenix, Focus) push finishes
12. Push Desktop to Beta at 100% from Ship-It via Ship RC.
    • 
    • Click the Schedule button.
      • 
13. Verify that the Balrog rule changes are live.
    1. View the Balrog Rules for Firefox Beta.
    2. Verify that the Background Rate for Firefox-xx.0-build1 is 100.
       • 
       • Please Note: The Fallback Mapping is the previous beta release.
14. Email release-signoff with a confirmation that updates are live.
    • See Example
15. Verify the package in the Microsoft Store submission is the correct version.
    • Please Note: The submission is automatically created as part of the RC build pipeline.
    • Verify that correct package was uploaded, the version displayed in the submission matches the release version
16. Submit the Desktop RC build to the Microsoft Store for review
    • Please Note: The Microsoft Store review can take several days, it’s recommended to submit for review at the end of RC week. If there is a blocker found later, you can always resubmit.
    1. Navigate to the Microsoft Partner Center
    2. Select Firefox from Windows & Xbox > Overview
    3. Select the pending submission
    4. In Packages enable gradual rollout and set to 25%
       • Ensure to enable the option to provide the newest package when users manually check for updates.
    5. In Submission Options set to Automatically Publish on the go-live date and time
    6. Submit to the store for review
    • Please see App submissions documentation for additional information on the Microsoft Store submission process.

The following tasks need to be performed at the start of RC week (week before go-live) for iOS

Firefox/Focus/Klar iOS

1. Check if there are any missing backports from the previous release.
   • Ensure that any backports from the dot releases on the previous version were also backported.
2. Check if there are any pending PRs for the firefox-ios release branch.
   • firefox-ios
   • Example Filter: is:pr is:open base:release/v115
3. Sync with the iOS team for Hard Freeze.
   • Ask on #firefox-ios-releases Slack channel
   • See Example
4. Run the Firefox:import translations (strings sync) action, set the target branch to the release branch (ex: release/v118).
   • See documentation here.
5. Trigger the firefox-ios SPM_Deploy_Prod_Beta bitrise build workflow for the release branch.
6. Trigger the firefox-ios focus_Release bitrise build workflow for the release branch.
7. Verify that Firefox iOS release is available in TestFlight.
   • Firefox TestFlight
8. Verify the Focus/Klar release is available in TestFlight.
   • Firefox Focus TestFlight
   • Firefox Klar TestFlight

As available, the following should be monitored/performed during RC week for Desktop/Android

1. Phabricator patches for remote-settings updates are generated automatically. Typically this is every Monday and Thursday. These patches must be included in release/esr uplifts.
   • Monitor email notifications sent by Phabricator: “No Bug, mozilla-release repo-update HSTS HPKP remote-settings tld-suffixes”/“No Bug, mozilla-esr91 repo-update HSTS HPKP remote-settings tld-suffixes”
   • Review the patch.
   • Commit the patch to release/esr via the moz-phab tool
     • moz-phab patch <patch id> --no-bookmark --apply-to here
   • Edit the commit message to add the approval.
2. Update the Weekly Digest document, as requested on #fx-weekly-digest.
   • Update the Release Milestones (Details) section to add Status (On Track, Delayed), Ship Date.
   • Review the Release Milestones section dates and update the status.
3. Verify that QA have signed off manual testing for Desktop
   • QA will post a message to the #qa-coordination Slack channel when they complete update testing and another message when they complete functional testing.
   • Build validation testing is tracked here
4. Verify that QA have signed off update tests on Beta.
   • QA will post a message to the #qa-coordination Slack channel when they complete update testing and another message when they complete functional testing.
5. Verify that QA have signed off update tests on release-localtest
   • QA will post a message to the #qa-coordination Slack channel when they complete update testing and another message when they complete functional testing.
6. Monitor for QA sign-off on Fenix/Focus build validation before proceeding with Steps 7 to 12.
   • Please Note: Build Validation sign-off is usually provided the day after Fenix/Focus builds are produced.
   • Focus and Fenix QA sign-off are sent via #qa-coordination Slack Channel.
7. Roll out Fenix to the Production track at 5% in google play
   • See Firefox Fast & Private Browser
   • Reply to the QA sign-off slack message
     • “Thanks, pushed to Production at 5% rollout.”
8. Roll out Focus/Klar to the Production track at 5% and Open Testing track at 100% in google play
   • See Firefox Focus: No Fuss Browser and Firefox Klar Browser
   • Reply to the QA sign-off slack message
     • “Thanks, pushed to Production at 5% rolled out.
9. Upload Fenix to Samsung Store with 5% rollout when approved
   • Upload ARM32 and ARM64 APKs (armeabi-v7a/arm64-v8a)
     • The APKs are available as an artifact in the signing-apk job from the Promote graph.
   • Set the rollout rate to 5%
   • Set to publish automatically
   • Please note: It can take up to 2 days for review.
10. Upload Focus to Samsung Store with 5% rollout when approved
    • Perform the same steps as Step 9

As available, the following should be monitored/performed during RC week for iOS

Focus iOS

1. Monitor for QA sign-off on Focus iOS build validation before proceeding with Steps 2 and 3.
   • Please Note: Build Validation sign-off is usually provided the day after the builds are produced.
   • Focus iOS QA sign-off is sent via #qa-coordination Slack Channel.
2. Submit iOS Focus and iOS Klar in the AppStore for review.
   • Select a phased rollout when submitting.
   • Select to automatically release after App Store Review on Sunday night Eastern prior to go-live (i.e. go-live date -2)
     • Note: If the Monday of go-live week is a Canadian public holiday, then set the automatic release date to Monday night Eastern instead.
3. Tag the release in GitHub in the firefox-ios repo using the format `focus/klar-vXXX.X`
   • Wait until the Friday of RC week to avoid tagging if another RC is needed
   • firefox-ios/releases
   • See Example

Firefox iOS

1. Monitor for QA sign-off on Firefox iOS build validation before proceeding with Steps 2 to 6.
   • Please Note: Build Validation sign-off is usually provided the day after the builds are produced.
   • Firefox iOS QA sign-off is sent via #qa-coordination Slack Channel.
2. Add the External Beta Testers group to the Firefox Beta TestFlight Build
3. Gather Firefox iOS release notes.
   • This can be done in parallel to the Desktop/Android release note gathering.
   • The store release notes are needed before submitting.
4. Submit Firefox iOS in the AppStore for review.
   • Select a phased rollout when submitting.
   • Select to automatically release after App Store Review on Sunday night Eastern 10PM (Monday 2AM GMT) prior to go-live (i.e. go-live date -2)
     • Note: If the Monday of go-live week is a Canadian public holiday, then set the automatic release date to Monday night Eastern instead.
5. Tag the release in GitHub in the firefox-ios repo using the format `firefox-vXXX.X`
   • Wait until the Friday of RC week to avoid tagging if another RC is needed
   • firefox-ios/releases
   • See Example

General iOS tasks

1. Bump the Firefox iOS version in the release branch to XXX.1
   • Wait until the Friday of RC week to avoid bumping the version if another RC is needed

RC Uplifts

This tab is for tracking bugs that are being tracked for possible uplift to the mozilla-release repository for RC builds. The primary objectives are:

• Track whether there are any drivers for a respin of the RC builds during RC week.
• Assess whether Desktop, Mobile, or both are affected by the issues noted.
• Verify that all drivers have had an explicit decision made.

The following tasks need to be performed during RC week (the week before go-live)

1. Monitor Uplift Requests for Beta/Release
   • For Example
   • Track if there are any drivers for an RC respin during RC week.
   • Assess whether Desktop, Mobile, or both are affected by the issues noted.
   • Verify that all drivers have had an explicit decision made.
2. If a new RC is required, for example, a release blocker was identified, then create a new RC checklist tab and perform the same steps as for the initial RC but with the following exceptions:
   1. Cancel the previous RC build in ship-it
   2. Delete the Microsoft Store submission before creating a new RC build
      • Please Note: The RC build will fail if trying to create an MSIX submission when there is already a pending submission
      • To cancel and delete the pending submission you may first need to change it from publishing automatically to publishing manually
3. Determine if a new RC build impacts Android.
   • If a new Android build is not required, then there is no impact on the Fenix/Focus builds and no action is required.
   • If a new Android build is required, then repeat the RC steps with the following exceptions:
     • Cancel the previous Firefox Android (AC, Focus) build in Ship-It.
     • Cancel the previous Fenix build in Ship-It.
     • If Fenix/Focus/Klar already rolled out - Replace the 5% rollout of Fenix/Focus/Klar in Google Play and the Samsung Store.

Go-Live Checklist

Once RC week is over and the release candidate is ready to go-live, preparations for go-live begin. After go-live, monitoring continues and if needed preparing for potential dot releases.

The following tasks need to be performed prior to go-live

1. Ensure that feedback for release notes was gathered.
   • Monitor the #release-notes Slack channel
   • For more information on the release notes doc process see here
2. Run the new-contributor.py script and add to the release notes
   • For more information on the release notes doc process see here
3. Release notes RelMan peer review.
   • For more information on the release notes doc process see here
4. Incorporate feedback into draft release notes.
   • Add a release in Nucleus for Firefox, Firefox for Android, Firefox for iOS
   • Duplicate the previous release, edit the version, dates, and remove the previous release notes. For example Firefox, Firefox for Android, Firefox for iOS
   • Create release notes entries as outlined in the document shared on the #release-notes Slack channel
   • Please Note: If images are included with the release note then create a PR for Bedrock, example PR
   • Share a link to the staged release notes on #release-notes
     • Example
5. Ensure that Legal signoff on release notes was requested by UX/Product
   • Please Note: This step is not always required, if the release notes are not atypical
6. Review tracking-firefoxXX+ bugs and confirm there are no blocking issues
7. Check for crash spikes with RC builds
8. Schedule push to CDN from Ship-It
   • Click Push
   • 
   • Click Schedule
   • 
9. Ensure the automated email was sent after the push to release-cdntest
   • Taskcluster email: “firefox xx.0 buildx/mozilla-release has been pushed to cdntest.
10. Ensure QA have performed the Update test on release-cdntest
    • QA will post a message to the #qa-coordination channel in Slack when they complete update testing and another message when they complete functional testing.
    • Build validation testing is tracked here
11. Ensure Legal have signed off on release notes
    • Please Note: This is only required based on the outcome of Step 3.
12. Ensure that the security advisories are ready
    • Check with the security team on the #security Slack channel

The following tasks need to be performed on go-live day

1. Schedule push to release at 25% from Ship-It
2. Bump Fenix rollout rate in the play store to 25%
   • See Firefox Fast & Private Browser
3. Bump Focus/Klar rollout rate in the play store to 25%
   • See Firefox Focus: No Fuss Browser and Firefox Klar Browser
4. Bump Fenix rollout rate in the Samsung Store to 25%
5. Bump Focus rollout rate in the Samsung Store to 25%
6. Ship Firefox Android (AC, Fenix, Focus) from Ship-It via Ship
7. Verify that the Microsoft Store is published at 25% rollout
8. Make release notes live in Nucleus
   • Please Note: There is a 15-30min lag between making the change in Nucleus and the live website picking up this change, so plan to do this 15-30min prior to go-live.
9. Upload security advisories to GitHub
10. Signoff on scheduled rule change in Balrog
    • Set the rollout percentage to 25%
    • Sync with release management for another team member to signoff on the rule change.
11. Send Slack message to #release-coordination to notify of the release
    • For example “Firefox Desktop 115.0, ESR 115.0.0, Fenix/Focus/Klar Android 115.0, and Firefox/Focus/Klar iOS 115.0 are live”
    • Include links to the release notes.
12. Verify that the new release is live on mozilla.org
13. Verify that the release notes are live here
14. Verify LATEST_FIREFOX_VERSION in firefox_versions.json
15. Verify that security advisories are live here
16. Ship new Desktop release in Ubuntu Snap Store
    • See Promote a snap to the stable channel for more information
17. Verify that the release is live on Flathub
    • The `release-flatpak-push-firefox` job runs as part of the ship graph and the publication is handled by Flathub.
    • https://flathub.org/api/v1/apps/org.mozilla.firefox contains the current release available on Flathub.
    • It can take up to 120 minutes for the release to publish on Flathub. If the release is not updated then reach out in the #flatpaks Matrix channel.
18. Email release-drivers & release-signoff that updates are live at 25%
    • Example Email
19. Ensure QA have performed the Update tests on Release
    • QA will post a message to the #qa-coordination Slack channel when they complete update testing and another message when they complete functional testing.
    • Build validation testing is tracked here
20. Email announce list
    • Example Email
21. Schedule Desktop update rate to 0% in Balrog after 24 hours.
22. Signoff on scheduled rule change in Balrog.
    • Sync with release management for another team member to signoff on the rule change.

The following tasks need to be performed 24 hours after go-live

1. Verify Desktop update rate at 0% in Balrog
2. Email release-signoff & release-drivers to confirm 0% throttling.
   • Reply to the previous email from the go-live day.
   • Example Email

The following tasks need to be performed 48 hours after go-live

1. Review release crash rates and incoming bugs for new blockers.
2. Bump Desktop update rate to 100% in Balrog.
3. Signoff on scheduled rule change in Balrog.
   • Sync with release management for another team member to signoff on the rule change.
4. Bump the Microsoft Store rollout to 100%.
   • Click Finalize package rollout
5. Review fenix/focus release crash rates, incoming bugs for new blockers, and spike/trend in negative play store reviews.
   • Check to see if there is anything concerning that would prevent completing the rollout.
6. Bump fenix/focus/klar rollout rate in the play store to 100%.
7. Bump fenix/focus rollout rate in the Samsung Store to 100%.
8. Email release-signoff & release-drivers to confirm full rollout.
   • Reply to the previous email from the go-live day.
   • Example Email

The following tasks need to be performed daily after go-live

1. Review release crash rates and incoming bugs for new blockers
2. Review uplift requests for dot release drivers and ride alongs

The following tasks need to be weekly after go-live

1. Update the Weekly Digest document, as requested on #fx-weekly-digest.
   • Update the Release Milestones (Details) section to add status on the release, dot release, etc.

Dot Release Uplifts

Similar to the RC Uplifts tab. The primary purpose of this tab is to track any bugs driving a dot release and bugs that are under consideration to ride-along, It is also used to assess which products are affected by any drivers. It is up to the release manager what should go into a dot release, and we try to keep these uplift guidelines in mind. Adding even what looks like a trivial fix can add risk, both to the process (delay, extra testing, work for other teams) and to causing new regressions. For security issues, we may not want to take them in a dot release unless there is particular pressure to do so.

The following information needs to be added for each bug:

• Bug number
• Component
• Is the bug a dot release driver?
• Does the patch affect Desktop, Mobile, or both?
• Is the fix ready?
• Is the patch on mozilla-central?
• Is there an uplift request on the ticket?
• Is the uplift approved?
• Is the patch uplifted to release?
• Does the bug need a release note?
• Any additional notes
  • For example, outline why the bug is being taken in a dot release.

Please Note: For each dot release, create a section in the Dot Release Uplifts tab to track the bugs in consideration: For example, “Desktop 101.0.1& Fenix/Focus 101.2.0 - GTB 2022-06-08”

Dot Release Checklist

The following tasks need to be performed at the start of preparing a dot release

1. Duplicate the Desktop Dot Release Checklist tab and name it appropriately.
   • Example: Desktop 101.0.1
2. Email release-drivers group to inform them a dot release is being created. #* Include information on the driver, if there will be an Android release, and a link to the sheet used to track the uplifts.
   • Example Email
3. Review any pending release uplift requests
   • Example Filter
   • If they are low risk consider including them in the dot release
4. Uplift patches to release.
5. Create a release in Nucleus to add release notes
   • Add a release note with a link to the major release for reference.
   • For bugs that have been identified as needing a release note:
     • If a release note is required and the release note text is clear from the bug content then create it directly in Nucleus.
     • Otherwise, request wording before adding the release note.
6. Follow the same process as creating an RC build
   • If any uplifts impact mobile
     • Duplicate the Mobile Dot Release Checklist tab and name it appropriately.
7. Follow the same process as reviewing an RC build has QA sign-off

The following tasks need to be performed during go-live of a dot release

1. Follow the same steps as an RC go-live, with the following exceptions:
   • The dot release is rolled out at 25% but there is no schedule drop to 0%
     • Please Note: Under chemspill situations, a greater rollout percentage may be required.
   • There may not be any security advisories.

The following tasks need to be performed 24 hours after go-live of a dot release

1. Follow the same steps as after an RC go-live, with the following exceptions:
   • If there are no blocking issues then bump the rollout to 100% after 24 hours

The following tasks need to be performed daily after go-live of a dot release

• Follow the same monitoring steps as after an RC go-live.