Sheriffing/How To/Intermittent bugs

How to file a Bug for a Test Failure aka Intermittent Bug

When you discover during your work on tree herder test failures who occurred only once (intermittent) we need a file a bug for this since the test might be need a change to make that test more stable.

In a lot of cases the test failure happened before and someone else or maybe yourself filed a bug for that failure days or weeks ago. In such cases treeherder suggest the Bug Number and title under the failure like

Filing a bug

Filing a Bug so that its getting picked up by treeherder as in the screenshot before is very easy.

Bascially 2 requirements need to be included in the bug that this bug can automatically picked up by treeheder:

1. In the summary: Intermittent test_file test failure 2. In the Keyword field choose the keyword: intermittent-failure

Best Pratice

Lets imagine you have a test failure like TEST-UNEXPECTED-TIMEOUT | /navigation-timing/test_timing_xserver_redirect.html | expected OK in Treeherder and there is no bug filed so far for this failure.

-> Open the Treeherder Log

1. Login into Bugzilla in a different tab
2. Find the Product/Component where you need to file this bug (note: mxr and hg.mozilla.org can be very helpful if you are in doubt)
3. In this example the Bug belong to Core:Document Navigation
4. Copy the failure text from the logwindow into the bug 
5. In the Summary: file the Bug as title: Intermittent navigation-timing/test_timing_xserver_redirect.html | expected OK
6. In the keyword field choose intermittent-failure
7. submit the bug

in a perfect world the bug would like https://bugzilla.mozilla.org/show_bug.cgi?id=1172135 :)

Treeherder syncs with bugzilla several times a day and so when this this is done from the next failure on your bug will be suggested for this failure!