Sheriffing/How To/Intermittent bugs

From MozillaWiki
< Sheriffing
Revision as of 12:19, 14 June 2017 by Tomcat (talk | contribs)
Jump to navigation Jump to search

How to file a Bug for a Test Failure aka Intermittent Bug

When you discover during your work on tree herder test failures who occurred only once (intermittent) we need a file a bug for this since the test might be need a change to make that test more stable.

In a lot of cases the test failure happened before and someone else or maybe yourself filed a bug for that failure days or weeks ago. In such cases treeherder suggest the Bug Number and title under the failure like

Treeherder suggestions

Filing a Bug so that its getting picked up by treeherder as in the screenshot before is very easy.

Bascially 2 requirements need to be included in the bug that this bug can automatically picked up by treeheder:

1. In the summary: Intermittent test_file test failure 2. In the Keyword field choose the keyword: intermittent-failure

Best Pratice

Lets imagine you have a test failure like TEST-UNEXPECTED-TIMEOUT | /navigation-timing/test_timing_xserver_redirect.html | expected OK in Treeherder and there is no bug filed so far for this failure.

-> Open the Treeherder Log 

1. Login into Bugzilla in a different tab
2. Find the Product/Component where you need to file this bug (note: mxr and hg.mozilla.org can be very helpful if you are in doubt)
3. In this example the Bug belong to Core:Document Navigation
4. Copy the failure text from the logwindow into the bug 
5. In the Summary: file the Bug as title: Intermittent navigation-timing/test_timing_xserver_redirect.html | expected OK
6. In the keyword field choose intermittent-failure
7. submit the bug

in a perfect world the bug would like https://bugzilla.mozilla.org/show_bug.cgi?id=1172135 :)

Treeherder syncs with bugzilla several times a day and so when this this is done from the next failure on your bug will be suggested for this failure!

Also if you want you can add %bugfiler to the treeherder url like https://treeherder.mozilla.org/#/jobs?repo=autoland&bugfiler - this activates bugfiler (the little bug icon beside a failure) that does most of the manual steps from the upper section

Retrieved from "https://wiki.allizom.org/index.php?title=Sheriffing/How_To/Intermittent_bugs&oldid=1173464"