Sheriffing/How To/Backouts

When a patch is landed that causes build or test failures, it must be backed out to restore the tree to a passing state again. This page documents the procedure for performing backouts both for Gecko changes and Gaia.

Gecko Backouts

Prerequisites

• Install the qbackout extension from https://hg.mozilla.org/hgcustom/version-control-tools/file/default/hgext/qbackout
• Instructions: https://hg.mozilla.org/hgcustom/version-control-tools/file/default/hgext/qbackout/README

Usage

Backout of a single changeset

hg qbackout -e -r 123456789abc

Where -r is the specific revision to be backed out. The -e option opens an editor to edit the commit message prior to committing. This is useful for adding DONTBUILD, CLOSED TREE, etc before pushing.

Backout of multiple consecutive changesets

hg qbackout -e -s -r 123456789abc:cba987654321

Where the -r is now specifying a range of commits (from 123456789abc to cba987654321). The -s option commits the backouts as a single commit. Otherwise, each changeset in the range will be backed out as separate commits.

Backout of multiple non-consecutive changesets

hg qbackout -e -s -r 123456789abc+cba987654321

Useful when there are commits in between the ones that need backing out.

Backout of a combination of consecutive and non-consecutive changesets

hg qbackout -e -s -r 123456789abc:cba987654321+xyz456789123

Useful when a follow-up patch needs backing out in addition to the main one (i.e. someone had a follow-up bustage fix push) or an additional commit conflicts with the backout and needs backing out as well.

Pushing

hg qfinish -a
hg push

Finalizes the patch for pushing to the remote repo.

What Ifs

• If the remote repo has additional changesets (i.e. a push race), hg pull --rebase will pull the new changesets and rebase the local changes on top of them.
  • This requires the rebase extension to be enabled in .hgrc
• If the commit message needs to be modifying after qfinish (i.e. forgotten CLOSED TREE), hg commit --amend will open the backout in an editor for modification.

Gaia Backouts

If the changeset is not part of a merge (i.e. pushed directly instead instead of a merged Pull Request):

git revert 12345ab

If the changeset is a merged Pull Request:

git revert -m 1 12345ab

Use |git push| once finished. Git also supports |pull --rebase| and |commit --amend| with the same usage as given above for Mercurial.

For reverting multiple git commits at the same time, see this StackOverflow post.

Best Practices

Based on Feedback from the Sheriff Survey, when you have to do a backout some best practice

• Ping the Developer in IRC if possible and when the developer is around, this gives the developer a chance to fix the problem and so avoid a backout. Some of the sheriffs use a 5 minute rule for the waiting of a response of the developer
• Set Need-info! This is very important! Developers get a lot of bugmail and so a backout comment can be missed - please make sure that you set a need-info to the developer in the bug when doing a backout
• Make sure that the failures that lead to the backout are stared in treeherder - this helps a lot also the following sheriffs.