B2G/QA/Automation/UI/Minimized Acceptance Execution
< B2G | QA | Automation | UI
Jump to navigation
Jump to search
This document is currently in work in progress
Current wording
- urgent job === smoke
- non-urgent job === others
Processing an automation report
- As we currently don't have automated email, you will need to fetch all the HTML report for every Jenkins job.
- Follow the
- Send out the email report.
Finding a failure in an automation report in an urgent job
General steps
- Immediately disable the test per the disable/xfail doc. This is the top priority.
- File or update a product bug if it reproduces manually, or an automation bug if it doesn’t, per the bug filing doc.
- Add the failure bug to the next standup highlights, at the standup notes
In details
Finding a failure in an automation report in a non-urgent job
- Immediately disable the test per the disable/xfail doc. This is the top priority.
- File or update a product bug if it reproduces manually, or an automation bug if it doesn’t, per the bug filing doc.
Starting work on a bug
- Assign yourself and fill the QA whiteboard to make it appear on the sprint page
- Investigate if it's either an automation issue or an infrastructure one.
- If the issue can be fixed, when your patch is ready, check Treeherder report before going any further.
- If not, nominate it for the flaky suite
- Ask 2 reviewers from either the core team or contributors. Only one is needed for enabling/disabling tests.
In details
Finishing work on a bug
- Either the second reviewer or you could ask Autolander to merge your patch.
Priorities
- Any urgent test failing due to automation issues should be fixed ASAP
- Any non-urgent, non-intermittent automation bug should be fixed within a week
- Any test that proves to be newly intermittent should be reviewed at next triage
- Any test failing more than 50% should be disabled and put on backlog
Transition plan
In order to get to the escalated jobs suite, here's how we'll proceed to reduce the daily human intervention:
- Reduction of the length of the automation report (see the new template)
- Configure Jenkins to send out email on every single failures (use this plugin to configure the content of the email.
- Stop to send out the manually written automation report. Product bugs will be covered in the general daily report.
- Split the jobs between flaky/non-flakies.