Compatibility/System Addon/Release Process

Interventions Releases

Module Owner: Dennis Schubert

Generally speaking, new interventions are landed in mozilla-central close to the start of each cycle's soft freeze period, and then ride the trains to Release. By essentially skipping the Nightly cycle, we speed up the process a bit, but we get some stability testing in Beta before the interventions are available to all users.

In cases that are more urgent, we can speed up the process by

• uplifting interventions into Beta at any time during the cycle,
• uplifting the interventions into Release to be shipped with the next dot-release,
• shipping an interventions add-on update out-of-band.

It is up to each release's Interventions Owner to decide if a WebCompat bug is important enough to warrant a fast process.

The normal process: Letting interventions ride the train

For low-priority interventions, the simplest way to ship is to land them in the tree and have them ride the trains like any code change.

1. Build and test the interventions as usual. Submit a Pull Request to the mozilla-extensions/webcompat GitHub repository that does include new interventions, but do not include a version number bump yet.
2. Bump the version number in manifest.json in your local clone of the repo, but do not commit the change yet.
3. Export your changed sources into your mozilla-central checkout. Check the addon's README.md for instructions on how to use the tooling.
4. Make sure ./mach lint browser/extensions/webcompat passes. Since the on-GitHub lints are not exactly the same (primarily, eslint is missing), it's a good idea to run that every time.
5. Submit the patch to Phabricator. You do not need to manually specify a reviewer: the webcompat-reviewers group will be added automatically, sending a notification to everyone in the team.
6. Land the patch. Wait until the patch got merged into mozilla-central to make sure it doesn't get backed out immediately.
7. Export the same sources into your local checkout of android-components. There is tooling to help you with that as well, see addon's README.md. Example Pull Request. To make it easier to keep track of, attach a link to the pull request to the corresponding bug on Bugzilla. If you don't know someone who can review your patch, ask in #fenix-team on Slack.
8. When the patch is merged, you can commit the version number change in the webcompat-addon repository. Make sure to run npm run autoformat before commiting, as that will apply your version number change to package.json and package-lock.json as well.
9. Tag the version with git tag -a v100.0.0 -m 'v100.0.0', and push both your version-bump-commit, and the new tag, to GitHub with git push origin main v100.0.0
10. Make sure that for any intervention you added, you set the sitepatch-applied label GitHub or the [webcompat:sitepatch-applied] whiteboard flag on Bugzilla. Likewise, remove those labels for all interventions you removed.

Uplifting new interventions into Beta

For interventions that are a bit more urgent, you can skip one release cycle and uplift your intervention(s) into Beta. However, do not just request beta uplift for the patch you submitted to mozilla-central:

• It is possible that your patch was not the only change to the addon since the Beta branch got merged. This would cause conflicts, because your new patch won't cleanly apply to Beta.
• Version numbers won't make sense: If you're uplifting a patch for the addon version 101.0.0 into Beta 100, you might make version numbers more confusing, and you also will make rolling out of-of-band updates a bit more complicated.
• It generally is a good idea to limit your Beta uplift to only the important/urgent things. It might not be a wise idea to uplift everything, which might include logic changes, SmartBlock Fixes, and other changes into beta. Keep the risk as low as possible.

Instead, we generally build a separate patch just for Beta.

1. Follow the "normal process", and get your interventions landed in mozilla-central. Wait for the next Nightly build, and make sure your interventions work as you expect.
2. In your local mozilla-central checkout, switch to the mozilla-beta branch/bookmark. If you've never done this before, check the documentation explaining how to work with the mozilla-unified repository.
3. Apply the relevant changes to the Firefox sources.
   • If your patch is simple, you can apply the changes directly to your mozilla-beta tree.
   • If your changes are more complicated, it might be a good idea to use the export tooling in our GitHub repo. You can git checkout the tag corresponding to the version currently shipping in Beta, and then git cherry-pick individual patches from main into your temporary branch. There is no need to publish/merge that branch upstream, as your beta rollout should never include any interventions that aren't also in the main branch anyway.
4. Publish the beta patch to Phabricator and have a webcompat-reviewer rubberstamp the patch. It is a good idea to add a clear indication into the commit message saying that this patch is meant for the Beta branch. Bug 1234567 - Beta - Uplift v101 Interventions into v100 Beta. is an example.
5. Set the approval-mozilla-beta? request flag, and explain in the comment why you think this patch should be uplifted to Beta.

A release manager will see your request. If your request is approved, the release managers will push your patch. You do not need to land it, or do anything else!

Uplifting new interventions into Release

If you want your interventions to be available to users very quickly, you can uplift them into mozilla-release and have it shipped in the next dot-release.

1. There is no fixed schedule for dot-releases, but you can reach out to the Release Owner and ask if a dot release is planned, and when it will be shipping. If there is a dot release coming up that fits your needs, let them know that you intend to ship WebCompat interventions in that release.
2. Get your interventions into mozilla-central first, using the "normal process" explained above. Make sure it works!
3. Then, you can follow the "uplifting into Beta" steps. The steps for Beta and Release are exactly the same, except that you're using a different branch/bookmark: mozilla-release instead of mozilla-beta.
4. You can request uplift for Beta and Release at the same time, but since these are two different patches, you have to request both uplifts separately in two comments.

Shipping interventions with an out-of-band System Addon Update

Prerequisites

Before you can do any of the work outlined here, you need to make sure that your Mozilla LDAP account is a member of the vpn_cloudops_shipit and xpi_system_build groups. If you are not sure, check your LDAP profile here (Mozilla-internal). If one or both groups are missing, file a bug similar to bug 1683143 (Mozilla-internal).

You also need to be able to connect to the Mozilla Corporate VPN to use ShipIt. See this guide (Mozilla-internal) if you're currently not set up for that.

Preparations

Since all changes that are about to be shipped need QA verification, we try to limit the changes to only the things we absolutely need to ship to Release users. As opposed to the other workflows, the primary work for rolling out an update happens on GitHub. Syncing the code back into mozilla-central is a step to be done after the update has been kicked off.

The addon's GitHub repository has a release-hotfix branch. This branch does not automatically track what's shipping in Firefox Release, and we do not have a process for updating the branch each release cycle. Instead, before doing any work, you have to update the hotfix branch to match the code shipping to Firefox Release. To update that branch:

1. Switch your local checkout of the addon sources to the release-hotfix branch. Make sure to git pull, in case your local checkout is outdated.
2. In your mozilla-central clone, check out the mozilla-release branch/bookmark.
3. Check which commit touched the landed addon sources last. If you're using Git for your mozilla-central, you can use git log browser/extensions/webcompat/. With git show [the last commit's hash] check which version number the sources were bumped to.
   • If the version number has a corresponding tag in the addon's repository, you can merge that tag into the release-hotfix branch with git merge --ff-only v100.2.1.
   • If the version number does not have a tag, check the addon's commit history, and find the commit hash of the backporting commit of that change. Merge the sources up until that commit into the release-hotfix branch with git merge --ff-only [commit hash].
   • If the change has not been backported, or you cannot merge a tag/commit cleanly without conflicts, press the "emergency meeting" button in the team. Something has gone really wrong.

If you're done, running the "export into mozilla-central" command should result in no actual changes to your mozilla-release checkout. If there are any changes, you merged the wrong tag or commit. Make sure the export is "clean" before proceeding. Otherwise, you might end up accidentally shipping the wrong/additional changes to release.

If the export is not resulting in any changes, git push the updated release-hotfix branch to GitHub.

Make sure the the Release Manager owning the target Firefox version is aware of your intentions to ship an update. Also, have a chat with our Softvision WebCompat team member so they can schedule a bit of time testing your release.

Preparing your patch

1. With the addon's release-hotfix branch as a base, switch to a new branch for your changes.
2. Write and test your interventions as usual. Commit them as usual as well.
3. In a separate commit in your branch, bump the version number by a patch-level increase, for example from 100.1.0 to 100.1.1. Commit that change as well. Make sure that the commit bumping the version number is last in your branch, if it isn't, use git rebase -i release-hotfix to reorder the commits.
4. Open a PR for your changes. GitHub should figure out that the base branch is release-hotfix, but if it doesn't, make sure to set that correctly before submitting the PR.
5. Have another member of the team review your PR.
6. Test your changes once again, for good measure. Then, merge your PR into the release-hotfix branch.

Kicking off the build process

For testing, we do not use the build artifacts for each commit on GitHub, and instead start a new release on ShipIt. The reason we do not use the dep-signed XPI from a commit's job is consistency: When you start a release on ShipIt, it builds an .xpi for you to test. In the end, once all testing and QA'ing is done, the same .xpi will be shipped to our users, just with a different signature. Essentially, this means we are QA'ing "the same binary" that will be shipped, reducing the chance of weird build-specific glitches.

1. Follow the "Starting the build phase" steps here.
   • The list of available XPIs has two WebCompat entires: webcompat and webcompat-release-hotfix. Pick webcompat-release-hotfix, as the other one will build and release the main branch.
   • When selecting the revision, double-check that the last revision is correct. This should be a merge commit merging your PR.
2. The "Build" link in ShipIt links to taskcluster. You'll see a dep-signing-webcompat-release-hotfix task. When the task is completed, the artifacts for that task includes a webcompat.xpi. Copy the URL to that.
3. Ask our QA friends to test that dep-signed XPI.
   • A template for a QA request can be found here.
   • The link to the XPI is the one you copied earlier.
   • Links to current unbranded builds can be found here.
4. After QA is done testing the dep-signed XPI, you need two System Addon admins (or one SAO admin and a Release Engineer as backup) to sign off. Use the #addons-pipeline channel on Slack to ask. To make everyone's work easy, provide a link to the bug and a link to the changes (for example a link to the merged GitHub PR).
5. When you have received the necessary signoffs, the Ship graph will be triggered, which will sign the .xpi with a Release certificate. Ask a Release Engineer (if in doubt, use the same Slack channel) to get the release-signed .xpi into Balrog's release-sysaddon channel. They'll need to know which Firefox version(s) you're targeting with your update, which usually is all minor versions of a given release, i.e. 100.*.
6. When a release engineer created and deployed the release-sysaddon rules, ask our QA people to run a second round of testing.
   • A template with all relevant explanations can be found here.
   • Note that this should be tested with the real Firefox Release version, not an unbranded build. This testing round is your final chance to catch any errors.
7. When your second QA run passed, ask Release Management for formal approval. This bug comment is an example. Provide as much detail as you can to make sure Release Management knows everything they need to know.
8. After Release Management has signed off, ask a Release Engineer to deploy the rule to the release channel.

The update is now live.

Follow-up work

• In the addon's GitHub repo, tag the commit you're rolling out. Push the tag, and merge the release-hotfix branch into main. This merge will most likely conflict, as the version numbers will disagree. Resolve that conflict by using the current version number from main and bump that number by one minor release.
• Even though your code is now shipping to release, please make sure to land your changes to mozilla-central, mozilla-beta, and mozilla-release. Nightly and Beta users will not receive your update, and even for Release users, it's important to ship the new addon version in a dot release, since new profiles otherwise won't get the new interventions immediately and have to wait for the background update. See the uplifting docs above, everything still applies.
• Make sure to label/close all relevant Bugzilla bugs, apply the sitepatch-applied label on GitHub, and all the other cleanup work you do in a regular release.
• Optional: You might be interested in keeping an eye on Telemetry data for the rollout. You can create a copy of this telemetry dashboard (Mozilla-internal), making sure to change the start date, the Firefox version, and the addon version as needed. If you are running into issues, ask :denschub, or ask in #data-help on Slack.

Interventions Release Rotations

Engineers on the Web Compatibility team will rotate on ownership of shipping new versions of our interventions addons, serving as an Intervention Release Owner (IRO). The process will follow a predictable 4 week schedule, mirroring the Firefox release schedule. However, we will attempt to land the interventions 1 week before soft freeze (see the schedule above for dates). Another way to look at this is 2 weeks before release.

A tracking bug for the next release should be filed in Bugzilla (in the Web Compatibility::Interventions component). Bugs for adding or removing interventions in the current release cycle should block this bug.

During bug diagnosis, if a site is identified as a low priority intervention candidate, a action-needssitepatch label is added for the next IRO to take care of during their rotation. High priority interventions should be flagged immediately to the IRO who will then begin the process necessary to ship an off-train intervention.

Candidates for interventions

There is a dedicated action-needssitepatch label on GitHub as well as a Bugzilla bug query for sites that need interventions. During a rotation, you should look at both of these sites and determine which are the most important to work on and ship (or close, if appropriate).

Upcoming Releases

• Version 104 - Tom - 2022-07-21
• Version 105 - Ksenia - 2022-08-18
• Version 106 - Dennis - 2022-09-15
• Version 107 - Dennis - 2022-10-13
• Version 108 - Tom - 2022-11-10
• Version 109 - Ksenia - 2022-12-08

Past Releases

• Firefox 74 (v7) - Ksenia
• Firefox 75 (v8) - Dennis
• Firefox 76 (v9) - Mike
• Firefox 77 (v10) - Ksenia
• Firefox 78 (v11) - Mike
• Firefox 79 (v12) - Dennis
• Firefox 80 (v13) - Karl
• Firefox 81 (v14) - Ksenia
• Firefox 82 (v15) - Tom
• Firefox 83 (v16) - Dennis
• Firefox 84 (v17) - Karl
• Firefox 85 (v18) - Ksenia
• Firefox 86 (v19) - Tom 2021-01-21 2021Q1
• Firefox 87 (v20) - Dennis 2021-02-18 2021Q1
• Firefox 88 (v21) - Karl 2021-03-18 2021Q1
• Firefox 89 (v22) - Ksenia 2021-04-15 2021Q2
• Firefox 90 (v23) - Tom 2021-05-27 2021Q2
• Firefox 91 (v24) - Dennis 2021-07-08 2021Q2
• Firefox 92 (v25) - Karl 2021-08-05 2021H2
• Firefox 93 (v26) - Ksenia 2021-09-02 2021H2
• Firefox 94 (v27) - Tom 2021-09-30 2021H2
• Firefox 95 (v28) - Dennis 2021-10-28 2021H2
• Firefox 96 (v29) - Karl 2021-12-02 2021H2
• Firefox 98 (v30) - Ksenia 2022-02-03 2022H1
• Firefox 99 (v31) - Tom 2022-03-03 2022H1
• Version 100 - Dennis 2022-03-31 2022H1
• Version 101 - Dennis 2022-04-28 2022H1
• Version 102 - Ksenia 2022-05-26 2022H1
• Version 103 - Dennis 2022-06-23 2022H1