Auto-tools/Projects/Stockwell/disable-recommended

This page is under construction!

[stockwell disable-recommended]

When an intermittent-failure bug has recorded 150 or more failures in the last 21 days, the Orange Factor robot will change the bug's stockwell whiteboard tag to "[stockwell disable-recommended]". These bugs should be reviewed at least twice per week (current schedule is Monday and Thursday).

This page describes how to review [stockwell disable-recommended] bugs.

Finding bugs to disable

• Use bugzilla to find all bugs with whiteboard tag "[stockwell disable-recommended]".
• For each disable-recommended bug, determine if the bug can be addressed by disabling tests.
  • Ignore "meta" bugs or any bugs not related to a particular test, some examples:
    • bug 1391545
    • bug 1407383
    • This may be a good time to "nag" the triage-owner for these bugs.
  • Eliminate bugs that have been recently fixed.
    • Check 7-day history on OrangeFactor.
    • bug 1420472
    • bug 1421775
    • bug 1423386 (fixed 2 days ago- recent comment and proved on graph)
    • If there are no recent failures, perhaps the bug can be marked [stockwell fixed], [stockwell disabled] or [stockwell unknown].
  • Eliminate bugs with a patch under review, or coming very soon.
    • Review recent bug history.
    • If there is a patch under review or comments indicate that a fix is coming soon, consider postponing action on this bug.

Finding the manifest

For each test to be disabled, find the associated test manifest:

• Search for the test name in searchfox or dxr, or,
• Run 'mach test-info --show-info <test-name>' in a mozilla-central check-out.

Determining which platforms are affected

On OrangeFactor, check the detail view for the bug for the last 30 days. Look down the list of bugs for affected Platforms and Build Types. Does this test need to be disabled on all platforms, or only on some? For opt or debug builds, or both?

Updating a manifest to disable a test

There are 3 different types of manifest files, each with a different format:

• "ini" manifests like mochitest.ini, chrome.ini, browser.ini, xpcshell.ini
• reftest or "list" manifests like reftest.list, crashtest.list
• web-platform manifests in testing/web-platform/meta

ini manifests

• Manifest example: https://searchfox.org/mozilla-central/source/browser/base/content/test/about/browser.ini

• In this type of manifest, add a "skip-if" statement under the test name -- on the next line.

[some_test.html]
skip-if = ...

• Some examples of "skip-if":

skip-if = true                             # skip this test everywhere / always (test never runs)
skip-if = os == "android"                  # skip on Android only (continues to run on Linux, Windows, etc.)
                                           # other os strings: "win", "linux", "mac"
skip-if = debug                            # skip on all debug builds
skip-if = !debug                           # skip on all non-debug builds (opt, pgo)
skip-if = os == "android" || os == "linux" # skip on Android and skip on Linux
skip-if = os == "android" && debug         # skip on Android/Debug only (continues to run on Android/Opt)

• other keywords: os_version, bits, e10s, stylo, webrender, asan
• Windows os versions: https://msdn.microsoft.com/en-us/library/windows/desktop/ms724832(v=vs.85).aspx

reftest manifests

• Manifest example: https://searchfox.org/mozilla-central/source/dom/canvas/test/reftest/filters/reftest.list
• In this type of manifest, add a skip-if() statement on the same line as the test:

 skip-if(...) == some_test.html some_ref.html

• Some examples of "skip-if":

skip-if(gtkWidget)                  # skip on Linux
skip-if(cocoaWidget)                # skip on Mac
skip-if(winWidget)                  # skip on Windows
skip-if(Android)                    # skip on Android
skip-if(isDebugBuild)               # skip on all Debug builds
skip-if(!isDebugBuild)              # skip on all non-Debug builds
skip-if(Android&&isDebugBuild)      # skip on Android/Debug

• Other keywords: stylo, webrender, oscpu
• Reference: https://searchfox.org/mozilla-central/source/layout/tools/reftest/README.txt#42

web-platform test manifests

(we haven't talked about these yet)

Making a patch

To actually disable a test, we need to update the manifest file in mozilla-central. Before doing that, create a mercurial patch containing the change, and upload the patch to bugzilla for review.

Some useful mercurial commands for creating a patch with "mq":

hg qseries       # see what patches are in your queue / what is applied
hg qdiff         # see the file changes in the current patch
hg qnew          # create a new patch
hg qrefresh      # update your patch with local file changes
hg qrefresh -e   # edit the commit message associated with your patch

Requesting review

In bugzilla, in the bug with [stockwell disable-recommended], use "Attach File" to attach your patch. Browse to find the patch in <your repo directory>/.hg/patches. Make sure you select "patch" for the Content Type. Then request review, "r?", and select a reviewer. In most cases, ask :gbrown or :jmaher for review.

There are additional, generic instructions for getting Mozilla reviews at:

https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/How_to_Submit_a_Patch

After the test is disabled

• Check OrangeFactor to see if the patch was effective!

Notes from email

Before disabling or when you don't disable:

• you have to read the history and see if anyone is working on it or trying to figure out the bug

* review recent bug history
** if there is a patch under review or comments indicate that a fix

is coming soon, consider postponing disabling

* look at the last 7 days in orange factor
** if no longer failing, mark [stockwell fixed]

• • some examples:
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1420472
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1421775
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1423386 (fixed 2 days ago- recent comment and proved on graph)

• if it is a harness (i.e. not a clear test case in the title), then comment and stress the importance
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1391545
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1407383

if there are tests to disable we need to edit the manifests:

• What type of test is this and where is the manifest? ./mach test-info
• Which platforms are affected? opt/debug/both, os, other
• One time mercurial setup:

https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Source_Code/Mercurial

• Creating a patch and requesting review

We'll need separate guides to disabling syntax for:

* manifest parser
** mochitest.ini, browser.ini, chrome.ini, xpcshell.ini, a11y.ini
* reftest manifests
** reftest.list, crashtests.list
** use fuzzing if possible
* web-platform tests
** testing/web-platform

• autophone (easily confused with above .ini/.list manifests)