Confirmed users
4,072
edits
Litmus:How to Triage Failed and Unclear test cases: Difference between revisions
Litmus:How to Triage Failed and Unclear test cases (view source)
Revision as of 16:54, 5 November 2008
, 5 November 2008no edit summary
mNo edit summary |
mNo edit summary |
||
| Line 13: | Line 13: | ||
= Review Failed results daily = | = Review Failed results daily = | ||
Once Failed results are found from the above formats, review each result by clicking the link to the result report. For each result reported: | Once Failed results are found from the above formats, review each result by clicking the link to the result report. For each result reported: | ||
* Confirm or deny the validity of the result. Results are automatically marked as Valid. It is up to the reviewer to determine which results are actually valid or invalid as Failed. | * Confirm or deny the validity of the result. Results are automatically marked as Valid. It is up to the reviewer to determine which results are actually valid or invalid as Failed. Results filed with an associated bug are likely to remain valid. Results filed with good comments should be closely reviewed, even tested to confirm validity. Results filed with no comments or associated bugs can be subjectively reviewed as to severity and known frequency of report. | ||
** If a result is valid as ''Failed'', make sure there is a bug on file for it in bugzilla.mozilla.org (check for one already filed, if not, create a new bug). Then leave the Valid?: box checked in the result report. | |||
** If the result is invalid, un-check the box in the Valid?: field of the result. | |||
* If the day of triage shows just a few valid reports, the lead can usually take on bug filing. However, if there are a lot of valid failed results, the Litmus lead should forward the results to the appropriate subgroup owner for them to take action. In which case, it will be their responsibility to file the bug then Vet the result. | * If the day of triage shows just a few valid reports, the lead can usually take on bug filing. However, if there are a lot of valid failed results, the Litmus lead should forward the results to the appropriate subgroup owner for them to take action. In which case, it will be their responsibility to file the bug then Vet the result. | ||
* Vet the result. Vetting is a way to mark results as having been reviewed and action taken. Once you are done with result validation and action taken, click the '''Vet Result''' button in the result report. | * Vet the result. Vetting is a way to mark results as having been reviewed and action taken. Once you are done with result validation and action taken, click the '''Vet Result''' button in the result report. | ||
| Line 24: | Line 23: | ||
= Review Unclear results at least once a week = | = Review Unclear results at least once a week = | ||
Once Unclear results are found from the above formats, review each result by clicking the link to the result report. For each result reported: | Once Unclear results are found from the above formats, review each result by clicking the link to the result report. For each result reported: | ||
* Confirm or deny the validity of the result. Results are automatically marked as Valid. It is up to the reviewer to determine which results are actually valid or invalid. | * Confirm or deny the validity of the result. Results are automatically marked as Valid. It is up to the reviewer to determine which results are actually valid or invalid. Results filed with good comments should be closely reviewed, even tested to confirm validity. Results filed with no comments can be subjectively reviewed as to severity and known frequency of report. | ||
** If a result is valid as ''Unclear'', the action to be taken is to edit the test case associated to the report and make it clear. | |||
** If the result is invalid, un-check the box in the Valid?: field of the result. | |||
* If the day of triage shows just a few valid reports, the lead can usually take correcting the test cases. However, if there are a lot of valid unclear results, the Litmus lead should forward the results to the appropriate subgroup owner for them to take action. In which case, it will be their responsibility to edit the test case to make it clear. | * If the day of triage shows just a few valid reports, the lead can usually take correcting the test cases. However, if there are a lot of valid unclear results, the Litmus lead should forward the results to the appropriate subgroup owner for them to take action. In which case, it will be their responsibility to edit the test case to make it clear. | ||
* Vet the result. Vetting is a way to mark results as having been reviewed and action taken. Once you are done with result validation and action taken, click the '''Vet Result''' button in the result report. | * Vet the result. Vetting is a way to mark results as having been reviewed and action taken. Once you are done with result validation and action taken, click the '''Vet Result''' button in the result report. | ||