B2G/QA/Automation/UI/Filing Bugs Against Automation Errors

Names and Errors

In templates below, it is very important to be consistent about <test name> and <error>. Being consistent about how these are supplied lets us easily search for whether a bug already exists for the failure.

<test name>

<test name> is at least the *.py filename. However, if the test function and file have different names, we need both. They can both be found in the HTML Report in Jenkins.

The HTML Report columns are:

• Error | filename.class | function

So, for example, if the HTML Report shows:

• Error | test_calendar_flick_through_months.TestCalendar | test_calendar_flick_through_months

...It's fine to just use test_calendar_flick_through_months.py, the filename.

However, if it shows:

• Error | test_browser_share_link.TestBrowserShareToMessages | test_browser_share_to_messages

...You'll want to use test_browser_share_link.py::test_browser_share_to_messages(), the filename and function name.

You do not need to include the class.

<error>

<error> is usually the last line of the stack trace. It will look something like:

AssertionError: False is not true

...or...

TimeoutException: TimeoutException: Timed out after 30.0 seconds with message: Plivo didn't report the call as completed

The most important part to include is the word with Error or Exception in it. If reasonable, include the whole line.

Searching for Bugs

As long as we stick with the above, you can easily search for open bugs by looking for bugs containing all the words <test name> and <error> (at least the Error/Exception part).

Jenkins Links

For automation bugs, if absolutely nothing else makes it into the bug, please at least include the link to the failed Jenkins job. Almost every other piece of information can be had or figured out from there.

IMPORTANT: Attaching the HTML report is not sufficient for this! The link lets us trace back to the build, look at console output, and other things way above and beyond the HTML Report.

If you copy/paste the requested information, there's no value in attaching the HTML report, so don't waste the time. Just include the link.

Templates for New Bugs

TODO: All sections except the first should also have information on how and when to proactively file a bug to disable/xfail the test in question.

Saw error, have no time to try to reproduce at all

This should be rare. We should try reproducing all failures from the report ASAP to check for product bugs. We'll put it on the backlog for tracking immediately, but it should be picked up urgently.

File bug in Gaia::UI Tests

• Summary: Triage <test name>, failed with <error>
• Product: Firefox OS
• Component: Gaia::UI Tests
• Platform: ARM
• OS: Gonk
• Keywords: none
• Whiteboard: none
• QA Whiteboard: [fxosqa-auto-backlog+]
• CC: None
• Body:
  • Link to the Jenkins HTML Report for failing job.
  • A statement that we haven’t tried reproducing manually yet and the result needs triage.
• Optional Body, but appreciated:
  • Copy/paste configuration block from top of Jenkins HTML Report
  • Traceback from test failure in the HTML Report

The next action here is to pick up this bug and attempt to reproduce, then change the bug as appropriate below.

Can not reproduce manually

This means we'll assume it's an automation bug. We'll put it on the backlog for tracking immediately. If it is a smoketest or sanity test bug, it should be picked up urgently. Otherwise, it should be triaged for failure rate.

File bug in Gaia::UI Tests

• Summary: <test name> failed with <error>
• Product: Firefox OS
• Component: Gaia::UI Tests
• Platform: ARM
• OS: Gonk
• Keywords: none
• Whiteboard: none
• QA Whiteboard: [fxosqa-auto-backlog+]
• CC: none
• Body:
  • Link to the Jenkins HTML Report for failing job.
  • Copy/paste configuration block from top of that link
  • Traceback from test failure in the HTML Report
  • A statement that it couldn’t be reproduced manually
  • Any other notes you want to add regarding repro rate from local or ad hoc run, etc.

The next action here is to pick up this bug and debug the test, or triage it as failing infrequently enough to sit on for the moment.

Can reproduce manually, not known to be an intended product change

This means we'll assume it's a product bug.

File bug in appropriate product area, just as if found manually

• Use existing product bug template, but please add:
  • Whiteboard: [fromAutomation]
  • CC: gaia-ui-tests@boot2gecko.bugs
  • Body or Comment:
    • Found in <link to the Jenkins HTML Report for failing job>, also reproduced manually.

Can reproduce manually, known to be an intended product change

This means it's actually an automation bug due to not being in sync with the product.

File bug in Gaia::UI Tests

• Use template from Can not reproduce manually above.
• Include statement that it can be reproduced manually, but the product is correct and it's an automation bug.
• If it's easy to supply a bug number for the product change, please include that.