Sheriffing/How To/Backouts: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Overhaul the entire page)
(→‎Best Practices and Communication: add best way to ask developers if backout from mozilla-central and new Nightlies are needed)
 
(24 intermediate revisions by 5 users not shown)
Line 1: Line 1:
When a patch is landed that causes build or test failures, it must be backed out to restore the tree to a passing state again. This page documents the procedure for performing backouts both for Gecko changes and Gaia.
{{Sheriffing How To|Backouts}}
When a patch is landed that causes build or test failures, it must be backed out to restore the tree to a passing or green state again.  


= Gecko Backouts =
= How to perform backouts =
== Prerequisites ==
'''There are a few things to check out which can indicate the necessity of a backout:'''
* Install the qbackout extension from https://hg.mozilla.org/hgcustom/version-control-tools/file/default/hgext/qbackout
# there are ES (linting opt) failures: close the tree, backout, inform the developer
* Instructions: https://hg.mozilla.org/hgcustom/version-control-tools/file/default/hgext/qbackout/README
# Requested by the developer
# The push has many bustages
# The push has a high failure rate
# The failure occurs on multiple pushes
# The failure is not assigned to a bug
'''Before''' the backout, we should <span style="color:#FF0000">'''close the tree'''</span> from which we’ll backout in order to avoid race pushing.


== Usage ==
With unified repos, backouts are now easier to perform. Please follow the instructions for [[Sheriffing/How:To:SheriffingFromUnifiedRepos#Backouts|backing out from unified repos]].
=== Backout of a single changeset===
hg qbackout -e -r 123456789abc
Where -r is the specific revision to be backed out. The -e option opens an editor to edit the commit message prior to committing. This is useful for adding DONTBUILD, CLOSED TREE, etc before pushing.


=== Backout of multiple consecutive changesets ===
====='''Steps:'''=====
hg qbackout -e -s -r 123456789abc:cba987654321
# close the tree
Where the -r is now specifying a range of commits (from 123456789abc to cba987654321). The -s option commits the backouts as a single commit. Otherwise, each changeset in the range will be backed out as separate commits.
# <code>cd mozilla-unified</code> <span style="color:#14866d">//access the mozilla-unified directory</span>
# <code>hg pull <repo></code> <span style="color:#14866d">//download repo</span>
# <code>hg update <repo></code> <span style="color:#14866d">//update repo</span>
# identify the revision we want to backout
# <code>hg oops -er <revision></code> <span style="color:#14866d">//run the backout command, for a single changeset. If the backout impacts</span> <span style="color:#FF0000">'''multiple changesets'''</span>, use <code>hg oops -esr <revision>::<revision></code> <span style="color:#14866d">instead.<span style="color:#FF0000">If there are</span> <span style="color:#FF0000">'''two changesets with other unrelated changesets inbetween'''</span>, <span style="color:#14866d">use</span><code>hg oops -esr <revision>+<revision></code>
# open the text editor to add the description of the backout <span style="color:#FF0000">'''“Backed out changeset <revision> (bug X) for <insert reason here>”'''</span>. If the repo(s) are <span style="color:#FF0000">'''closed'''</span>, then we also need to add <span style="color:#FF0000">'''“on a CLOSED TREE”'''</span>. If the repo(s) are <span style="color:#FF0000">'''approval-only'''</span> we need to add <span style="color:#FF0000">'''a=backout'''</span>. The reason should mention the build or test suite (and in the latter case also the failing test) and the affected platforms if it doesn't fail on every.
# <code>hg push -r . <repo></code> <span style="color:#14866d">// push the changes</span>


=== Backout of multiple non-consecutive changesets ===
===== Example on how to backout a single revision =====
hg qbackout -e -s -r 123456789abc+cba987654321
'''Note:''' The example below is for autoland, but the same steps should be followed for any repos
Useful when there are commits in between the ones that need backing out.
# close the tree
# <code>cd mozilla-unified</code>
# <code>hg pull autoland</code>
# <code>hg update autoland</code>
# identify the revision we want to backout (e.g. def0af88e262)
# <code>hg oops -er def0af88e262 </code>
# open the text editor to add the description of the backout, e.g.: “Backed out changeset def0af88e262 (bug 1359017) for <insert reason here> CLOSED TREE”.
# <code>hg push -r . autoland</code>


=== Backout of a combination of consecutive and non-consecutive changesets ===
===== Example on how to backout multiple revisions =====
hg qbackout -e -s -r 123456789abc:cba987654321+xyz456789123
'''Note:''' The example below is for autoland, but the same steps should be followed for any repos
Useful when a follow-up patch needs backing out in addition to the main one (i.e. someone had a follow-up bustage fix push) or an additional commit conflicts with the backout and needs backing out as well.
# close the tree
# <code>cd mozilla-unified </code>
# <code>hg pull autoland </code>
# <code>hg update autoland </code>
# identify the push we want to backout. E.g.: 478cffed4b5f, daea2bcda8cb, 9abaf5195566, 2f665a0a379f
# <code>hg oops -esr  2f665a0a379f::478cffed4b5f </code>
# open the text editor to add the description of the backout, e.g.: “Backed out X changesets (bug 1359017) for <insert reason here> CLOSED TREE”.
# <code>hg push -r . autoland </code>


== Pushing ==
= Best Practices and Communication =
hg qfinish -a
Based on Feedback from the Sheriff Survey, when you have to do a backout, here are some best practices:
Finalizes the patch for pushing to the remote repo.
* Ping the Developer in IRC if possible and when the developer is around, this gives the developer a chance to fix the problem and so avoid a backout. Some of the sheriffs use a 5 minute rule for the waiting of a response of the developer. Note: this is done as a courtesy to developers. You can and should backout patches if you feel it's being abused, e.g. the developer is not using the [[Sheriffing/How:To:Recommended_Try_Practices|Try server]]
* After you backed a change out, update the bug in Bugzilla:
** Mention the reason for the backout.
** Link to the backout commit starting with https://hg.mozilla.org
** Link to the push which got backed out, use a link which also shows classified failures. That way developers can check what platforms are affected and if there are more issues.
** Link to failure log of a failed log for easy access.
** Snippet of the failure message: This might provide insight if the developer can fix it quickly.
** Set the NEEDINFO flag in bugzilla. This is very important! Developers get a lot of bugmail and backout comments can be missed. Please make sure that you set a NEEDINFO to the developer in the bug when doing a backout. This is usually done by selecting "assignee" from the needinfo dropdown near the bottom.
* Make sure that the failures that lead to the backout are [[Sheriffing/How:To:Treeherder#Classifying.2FStarring_jobs|starred in Treeherder]]. This helps the next sheriff on duty.
* First a change might be backed out for one issue but later a longer running task also fails due to those changes but maybe with a completely unrelated test. Update the backed out bug about this. Use a new comment, don't edit the previous backout comment because there is no bug mail for that.


=== What Ifs ===
==Backouts from mozilla-central==
* If the remote repo has additional changesets (i.e. a push race), hg pull --rebase will pull the new changesets and rebase the local changes on top of them.
If a developer asks for a backout of patch which has already been merged to central, code sheriffs should ask them if the fix requires to ship a new Nightly as soon as possible. If yes, back it out from central, else request new Nightlies and merge back to autoland, else just back it out from autoland. This guides the developer better than the question if the backout should be done on autoland or central.
** This requires the [http://mercurial.selenic.com/wiki/RebaseExtension rebase] extension to be enabled in .hgrc
* If the commit message needs to be modifying after qfinish (i.e. forgotten CLOSED TREE), hg commit --amend will open the backout in an editor for modification.


= Gaia Backouts =
= How to test a backout on the Try server =
If the changeset is not part of a merge (i.e. pushed directly instead instead of a merged Pull Request):
If there is uncertainity if a backout will fix an issue, this can be tested on the [https://wiki.mozilla.org/ReleaseEngineering/TryServer Try server] where changes are tested but never committed to production repositories.
git revert 12345ab
<ol>
<li>First backout the changeset(s)<br/>
<code>hg oops -er</code> or <code>hg oops -esr</code></li>
<li>At https://mozilla-releng.net/trychooser/ choose the build types, the platforms, test suites etc. that you need to test.<br/>
Example:<br/>
[[File:Sheriffing-Try server config example.png]]
</li>
<li>For '''the backout’s commit message''', we will use what is written in the '''Computed Syntax''' command field. Example:<br/>
[[File:Sheriffing-try server commit message.png]]
</li>
<li>To push to try, use the following command:<br/>
<code>hg push -f ssh://hg.mozilla.org/try</code>
</li>
<li>After the backout was pushed to Try, we need to drop the commit from history, for every single changeset that was tested:<br/>
'''Run hg histedit and replace pick with d.'''</li>
</ol>


If the changeset is a merged Pull Request:
= Rejected backout pushes =
git revert -m 1 12345ab
By default, the server will reject changes to some folder or file types and a keyword needs to be part of the commit message to let it pass. This has also been used when the change landed (can also be in a different line of the commit message than the first one - in that case Treeherder won't show it) and is usually written in UPPERCASE.


For reverting multiple git commits at the same time, see [http://stackoverflow.com/questions/1463340/revert-multiple-git-commits this StackOverflow] post.
Examples:
* <code>changesets contain changes to protected nsprpub/ directory</code>: Commit message requires <code>UPGRADE_NSPR_RELEASE</code>
* <code>changesets contain changes to protected security/nss/ directory</code>: Commit message requires <code>UPGRADE_NSS_RELEASE</code>
 
= Useful commands =
If you want to find all changesets which landed for a bug number, e.g. in multiple commits:
 
<code>hg log -k "bug number"</code>
 
E.g.
 
<code>hg log -k "1234567"</code>

Latest revision as of 13:13, 9 August 2022

When a patch is landed that causes build or test failures, it must be backed out to restore the tree to a passing or green state again.

How to perform backouts

There are a few things to check out which can indicate the necessity of a backout:

  1. there are ES (linting opt) failures: close the tree, backout, inform the developer
  2. Requested by the developer
  3. The push has many bustages
  4. The push has a high failure rate
  5. The failure occurs on multiple pushes
  6. The failure is not assigned to a bug

Before the backout, we should close the tree from which we’ll backout in order to avoid race pushing.

With unified repos, backouts are now easier to perform. Please follow the instructions for backing out from unified repos.

Steps:
  1. close the tree
  2. cd mozilla-unified //access the mozilla-unified directory
  3. hg pull <repo> //download repo
  4. hg update <repo> //update repo
  5. identify the revision we want to backout
  6. hg oops -er <revision> //run the backout command, for a single changeset. If the backout impacts multiple changesets, use hg oops -esr <revision>::<revision> instead.If there are two changesets with other unrelated changesets inbetween, usehg oops -esr <revision>+<revision>
  7. open the text editor to add the description of the backout “Backed out changeset <revision> (bug X) for <insert reason here>”. If the repo(s) are closed, then we also need to add “on a CLOSED TREE”. If the repo(s) are approval-only we need to add a=backout. The reason should mention the build or test suite (and in the latter case also the failing test) and the affected platforms if it doesn't fail on every.
  8. hg push -r . <repo> // push the changes
Example on how to backout a single revision

Note: The example below is for autoland, but the same steps should be followed for any repos

  1. close the tree
  2. cd mozilla-unified
  3. hg pull autoland
  4. hg update autoland
  5. identify the revision we want to backout (e.g. def0af88e262)
  6. hg oops -er def0af88e262
  7. open the text editor to add the description of the backout, e.g.: “Backed out changeset def0af88e262 (bug 1359017) for <insert reason here> CLOSED TREE”.
  8. hg push -r . autoland
Example on how to backout multiple revisions

Note: The example below is for autoland, but the same steps should be followed for any repos

  1. close the tree
  2. cd mozilla-unified
  3. hg pull autoland
  4. hg update autoland
  5. identify the push we want to backout. E.g.: 478cffed4b5f, daea2bcda8cb, 9abaf5195566, 2f665a0a379f
  6. hg oops -esr 2f665a0a379f::478cffed4b5f
  7. open the text editor to add the description of the backout, e.g.: “Backed out X changesets (bug 1359017) for <insert reason here> CLOSED TREE”.
  8. hg push -r . autoland

Best Practices and Communication

Based on Feedback from the Sheriff Survey, when you have to do a backout, here are some best practices:

  • Ping the Developer in IRC if possible and when the developer is around, this gives the developer a chance to fix the problem and so avoid a backout. Some of the sheriffs use a 5 minute rule for the waiting of a response of the developer. Note: this is done as a courtesy to developers. You can and should backout patches if you feel it's being abused, e.g. the developer is not using the Try server
  • After you backed a change out, update the bug in Bugzilla:
    • Mention the reason for the backout.
    • Link to the backout commit starting with https://hg.mozilla.org
    • Link to the push which got backed out, use a link which also shows classified failures. That way developers can check what platforms are affected and if there are more issues.
    • Link to failure log of a failed log for easy access.
    • Snippet of the failure message: This might provide insight if the developer can fix it quickly.
    • Set the NEEDINFO flag in bugzilla. This is very important! Developers get a lot of bugmail and backout comments can be missed. Please make sure that you set a NEEDINFO to the developer in the bug when doing a backout. This is usually done by selecting "assignee" from the needinfo dropdown near the bottom.
  • Make sure that the failures that lead to the backout are starred in Treeherder. This helps the next sheriff on duty.
  • First a change might be backed out for one issue but later a longer running task also fails due to those changes but maybe with a completely unrelated test. Update the backed out bug about this. Use a new comment, don't edit the previous backout comment because there is no bug mail for that.

Backouts from mozilla-central

If a developer asks for a backout of patch which has already been merged to central, code sheriffs should ask them if the fix requires to ship a new Nightly as soon as possible. If yes, back it out from central, else request new Nightlies and merge back to autoland, else just back it out from autoland. This guides the developer better than the question if the backout should be done on autoland or central.

How to test a backout on the Try server

If there is uncertainity if a backout will fix an issue, this can be tested on the Try server where changes are tested but never committed to production repositories.

  1. First backout the changeset(s)
    hg oops -er or hg oops -esr
  2. At https://mozilla-releng.net/trychooser/ choose the build types, the platforms, test suites etc. that you need to test.
    Example:
    Sheriffing-Try server config example.png
  3. For the backout’s commit message, we will use what is written in the Computed Syntax command field. Example:
    Sheriffing-try server commit message.png
  4. To push to try, use the following command:
    hg push -f ssh://hg.mozilla.org/try
  5. After the backout was pushed to Try, we need to drop the commit from history, for every single changeset that was tested:
    Run hg histedit and replace pick with d.

Rejected backout pushes

By default, the server will reject changes to some folder or file types and a keyword needs to be part of the commit message to let it pass. This has also been used when the change landed (can also be in a different line of the commit message than the first one - in that case Treeherder won't show it) and is usually written in UPPERCASE.

Examples:

  • changesets contain changes to protected nsprpub/ directory: Commit message requires UPGRADE_NSPR_RELEASE
  • changesets contain changes to protected security/nss/ directory: Commit message requires UPGRADE_NSS_RELEASE

Useful commands

If you want to find all changesets which landed for a bug number, e.g. in multiple commits:

hg log -k "bug number"

E.g.

hg log -k "1234567"

Retrieved from "https://wiki.allizom.org/index.php?title=Sheriffing/How_To/Backouts&oldid=1243593"