canmove, Confirmed users
2,041
edits
B2G/QA/Automation/UI/Strategy/Streamline End to end Execution: Difference between revisions
Jump to navigation
Jump to search
B2G/QA/Automation/UI/Strategy/Streamline End to end Execution (view source)
Revision as of 04:11, 11 February 2015
, 11 February 2015no edit summary
(Created page with "=Strategy: Streamline UAT Execution= ==Objective== Reduce handoffs and increase alert automation in the UAT execution so that failures can be triaged more quickly and with l...") |
No edit summary |
||
| Line 11: | Line 11: | ||
* Test results only looked at a limited number of times per day | * Test results only looked at a limited number of times per day | ||
== The Problem == | |||
We've had too many moving parts in our execution process, and too much manual review. One team would examine the results and triage for automation failures. Things that didn't fail automation would be handed to a manual team, who'd triage for manual failures. | We've had too many moving parts in our execution process, and too much manual review. One team would examine the results and triage for automation failures. Things that didn't fail automation would be handed to a manual team, who'd triage for manual failures. | ||
| Line 19: | Line 17: | ||
None of this happened unless the results were explicitly looked at. Further, looking at the results is time-intensive because something always fails, so one needs to read them line-by-line to understand what is important. By the time we report bugs from automation, they've frequently already been found through some other process. | None of this happened unless the results were explicitly looked at. Further, looking at the results is time-intensive because something always fails, so one needs to read them line-by-line to understand what is important. By the time we report bugs from automation, they've frequently already been found through some other process. | ||
== The Solution == | |||
High-urgency UAT runs such as smoketests will be aggressively pruned for unreliable tests; these will be moved into their own test runs. Once high-urgency runs are stabilized they will enhanced to alert proactively via email to QA or a similar mechanism. An alert is an unambiguous indication to immediately test manually and file any necessary bugs. | High-urgency UAT runs such as smoketests will be aggressively pruned for unreliable tests; these will be moved into their own test runs. Once high-urgency runs are stabilized they will enhanced to alert proactively via email to QA or a similar mechanism. An alert is an unambiguous indication to immediately test manually and file any necessary bugs. | ||
| Line 27: | Line 25: | ||
Of the remaining tests that cannot be reliably alerted upon, they will be triaged as they are now, but immediately tested manually by the triaging team with any product bugs immediately filed and escalated if necessary. | Of the remaining tests that cannot be reliably alerted upon, they will be triaged as they are now, but immediately tested manually by the triaging team with any product bugs immediately filed and escalated if necessary. | ||
== Timeline == | |||
Reduce Handoffs | Reduce Handoffs | ||