Sheriffing/How To/Intermittent bugs: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
No edit summary
Line 1: Line 1:
= How to file a Bug for a Test Failure aka Intermittent Bug =
When you find a test that fails more than once, but not every time, congratulations, you've uncovered an intermittent failure. This is the most annoying class of failures for both sheriffs and developers because it is not necessarily related to the code under test, but more likely indicates that the test itself might need to change to improve stability.


When you discover during your work on tree herder test failures who occurred only once (intermittent) we need a file a bug for this since the test might be need a change to make that test more stable.
The test failure may have happened before and a bug may already be on file. In such cases, Treeherder should suggest the bug number and title under the failure, e.g.:
 
In a lot of cases the test failure happened before and someone else or maybe yourself filed a bug for that failure days or weeks ago. In such cases treeherder suggest the Bug Number and title under the failure like


[[File:Treeherder suggestion.png|Treeherder suggestions]]
[[File:Treeherder suggestion.png|Treeherder suggestions]]


Filing a Bug so that its getting picked up by treeherder as in the screenshot before is very easy.
= How to file a bug for an intermittent failure =


Bascially 2 requirements need to be included in the bug that this bug can automatically picked up by treeheder:
If there's no bug on file, you'll need to file one. The are two requirements need to be included in the bug that this bug can be displayed automatically by Treeheder when this intermittent failure happens again:


1. In the summary: '''Intermittent''' test_file test failure
# In the summary: '''Intermittent''' test_file test failure
2. In the Keyword field choose the keyword: '''intermittent-failure'''
# In the Keyword field choose the keyword: '''intermittent-failure'''


== Best Pratice ==
== Example ==


Lets imagine you have a test failure like TEST-UNEXPECTED-TIMEOUT | /navigation-timing/test_timing_xserver_redirect.html | expected OK in Treeherder and there is no bug filed so far for this failure.  
Lets imagine you have a test failure like <code>TEST-UNEXPECTED-TIMEOUT | /navigation-timing/test_timing_xserver_redirect.html | expected OK</code> in Treeherder and there is no bug on file for this failure.  


-> Open the Treeherder Log
# Open the Treeherder Log
1. Login into Bugzilla in a different tab
# Login into Bugzilla in a different tab/window
2. Find the Product/Component where you need to file this bug (note: mxr and hg.mozilla.org can be very helpful if you are in doubt)
# Find the Product/Component where you need to file this bug (note: dxr and hg.mozilla.org can be very helpful if you are in doubt)
3. In this example the Bug belong to Core:Document Navigation
#* In this example the Bug belong to Core:Document Navigation
4. Copy the failure text from the logwindow into the bug  
# Copy the failure text from the log window into the bug  
5. In the Summary: file the Bug as title: Intermittent navigation-timing/test_timing_xserver_redirect.html | expected OK
# Set the Summary as: Intermittent navigation-timing/test_timing_xserver_redirect.html | expected OK
6. In the keyword field choose intermittent-failure
# In the keyword field choose intermittent-failure
7. submit the bug
# Submit the bug


in a perfect world the bug would like https://bugzilla.mozilla.org/show_bug.cgi?id=1172135 :)
The bug should look like https://bugzilla.mozilla.org/show_bug.cgi?id=1172135


Treeherder syncs with bugzilla several times a day and so when this this is done from the next failure on your bug will be suggested for this failure!
Treeherder syncs with Bugzilla several times a day. Once your bug is added and the systems sync, Treeherder will suggest your new bug as a match for the next intermittent failure of this type.


Also if you want you can add %bugfiler to the treeherder url like https://treeherder.mozilla.org/#/jobs?repo=autoland&bugfiler - this activates bugfiler (the little bug icon beside a failure) that does most of the manual steps from the upper section
= Bugfiler =
You can add <code>%bugfiler</code> to the Treeherder url, e.g. https://treeherder.mozilla.org/#/jobs?repo=autoland&bugfiler This activates bugfiler (the little bug icon beside a failure) that automates most of the manual steps from the previous section.

Revision as of 16:08, 21 July 2017

When you find a test that fails more than once, but not every time, congratulations, you've uncovered an intermittent failure. This is the most annoying class of failures for both sheriffs and developers because it is not necessarily related to the code under test, but more likely indicates that the test itself might need to change to improve stability.

The test failure may have happened before and a bug may already be on file. In such cases, Treeherder should suggest the bug number and title under the failure, e.g.:

Treeherder suggestions

How to file a bug for an intermittent failure

If there's no bug on file, you'll need to file one. The are two requirements need to be included in the bug that this bug can be displayed automatically by Treeheder when this intermittent failure happens again:

  1. In the summary: Intermittent test_file test failure
  2. In the Keyword field choose the keyword: intermittent-failure

Example

Lets imagine you have a test failure like TEST-UNEXPECTED-TIMEOUT | /navigation-timing/test_timing_xserver_redirect.html | expected OK in Treeherder and there is no bug on file for this failure.

  1. Open the Treeherder Log
  2. Login into Bugzilla in a different tab/window
  3. Find the Product/Component where you need to file this bug (note: dxr and hg.mozilla.org can be very helpful if you are in doubt)
    • In this example the Bug belong to Core:Document Navigation
  4. Copy the failure text from the log window into the bug
  5. Set the Summary as: Intermittent navigation-timing/test_timing_xserver_redirect.html | expected OK
  6. In the keyword field choose intermittent-failure
  7. Submit the bug

The bug should look like https://bugzilla.mozilla.org/show_bug.cgi?id=1172135

Treeherder syncs with Bugzilla several times a day. Once your bug is added and the systems sync, Treeherder will suggest your new bug as a match for the next intermittent failure of this type.

Bugfiler

You can add %bugfiler to the Treeherder url, e.g. https://treeherder.mozilla.org/#/jobs?repo=autoland&bugfiler This activates bugfiler (the little bug icon beside a failure) that automates most of the manual steps from the previous section.

Retrieved from "https://wiki.allizom.org/index.php?title=Sheriffing/How_To/Intermittent_bugs&oldid=1176256"