Auto-tools/Projects/Stockwell/disable-recommended

This page is under construction!

[stockwell disable-recommended]

When an intermittent-failure bug has recorded 150 or more failures in the last 21 days, the Orange Factor robot will change the bug's stockwell whiteboard tag to "[stockwell disable-recommended]". These bugs should be reviewed at least twice per week (current schedule is Monday and Thursday).

This page describes how to review [stockwell disable-recommended] bugs.

Finding bugs to disable

• Use bugzilla to find all bugs with whiteboard tag "[stockwell disable-recommended]".
• For each disable-recommended bug, determine if the bug can be addressed by disabling tests.
  • Eliminate "meta" bugs or any bugs not related to a particular test.
    • bug 1391545
    • bug 1407383
    • This may be a good time to "nag" the triage-owner for these bugs.
  • Eliminate bugs that have been recently fixed.
    • Check 7-day history on OrangeFactor: If there are no recent failures, perhaps the bug can be marked [stockwell fixed], [stockwell disabled] or [stockwell unknown].
    • bug 1420472
    • bug 1421775
    • bug 1423386 (fixed 2 days ago- recent comment and proved on graph)
  • Eliminate bugs with a patch under review, or coming very soon.
    • Review recent bug history.
    • If there is a patch under review or comments indicate that a fix is coming soon, consider postponing action on this bug.

After the test is disabled

• Check OrangeFactor to see if the patch was effective!

Notes from email

Before disabling or when you don't disable:

• you have to read the history and see if anyone is working on it or trying to figure out the bug

* review recent bug history
** if there is a patch under review or comments indicate that a fix

is coming soon, consider postponing disabling

* look at the last 7 days in orange factor
** if no longer failing, mark [stockwell fixed]

• • some examples:
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1420472
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1421775
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1423386 (fixed 2 days ago- recent comment and proved on graph)

• if it is a harness (i.e. not a clear test case in the title), then comment and stress the importance
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1391545
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1407383

if there are tests to disable we need to edit the manifests:

• What type of test is this and where is the manifest? ./mach test-info
• Which platforms are affected? opt/debug/both, os, other
• One time mercurial setup:

https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Source_Code/Mercurial

• Creating a patch and requesting review

We'll need separate guides to disabling syntax for:

* manifest parser
** mochitest.ini, browser.ini, chrome.ini, xpcshell.ini, a11y.ini
* reftest manifests
** reftest.list, crashtests.list
** use fuzzing if possible
* web-platform tests
** testing/web-platform

• autophone (easily confused with above .ini/.list manifests)