Confirmed users
14,525
edits
QA/Platform/DOM/Metrics: Difference between revisions
→Phase I: Establishing a Baseline
| Line 8: | Line 8: | ||
== Phase I: Establishing a Baseline == | == Phase I: Establishing a Baseline == | ||
''The goal of the initial phase is to establish a baseline that all future quality can be measured against.'' In implementation we will experiment with a lot of different metrics and data sources in an attempt to see which metrics, either singularly or in conjunction with other data, enable us to effectively make informed decisions. | ''The goal of the initial phase is to establish a baseline that all future quality can be measured against.'' | ||
In implementation we will experiment with a lot of different metrics and data sources in an attempt to see which metrics, either singularly or in conjunction with other data, enable us to effectively make informed decisions. | |||
; Catalog of Potential Data Sources and Metrics | |||
* https://github.com/AutomatedTester/futurama-data | |||
** intermittent test failure rate | |||
** backout rate | |||
* http://brasstacks.mozilla.com/testreports/weekly | |||
** tests enabled week over week | |||
** tests disabled week over week | |||
** tests skipped week over week | |||
* https://bugzilla.mozilla.org | |||
** crash bugs over time ([https://drive.google.com/open?id=1T0ZMPtojYsWcd1oNmCVypV9cJgnnx-VYcDv7P0t945c&authuser=1 draft]) | |||
** intermittent failure bugs by age ([https://drive.google.com/open?id=1ZaRcZGpti2Ti8-5FQybwFsRdvG7PdO5K9aw9SYpG-Rs&authuser=1 draft]) | |||
** intermittent failure bugs by number of comments ([https://drive.google.com/open?id=1ZRWaQ_XfH8x8THUJ8G3oTHvgWpjRm6HpaIbFwZCkYaA&authuser=1 draft]) | |||
** intermittent failure bugs by sub-component ([https://drive.google.com/open?id=1NehtD8rD6R_K89Ly9S63nIWAWeSsOexa-zANgF4OXCs&authuser=1 draft]) | |||
** intermittent failure bugs in/out ([https://drive.google.com/open?id=1bTNIeMCwgDm9kV8u6Z-DhxOnBiynGpAN6c6QdNBoNXQ&authuser=1 draft]) | |||
** regression bugs by age ([https://drive.google.com/open?id=1ooYMqeGe0xjcSYS1YRogEdGItKRoPFVqJDlshpL-iYQ&authuser=1 draft]) | |||
** regression bugs by number of comments ([https://drive.google.com/open?id=1ZZL7WU3H5h9sM_1XVJ1HF7oD-s4q3EwSy-sGu9NLeOg&authuser=1 draft]) | |||
** regression bugs in/out ([https://drive.google.com/open?id=1NzjmQP7Io2rWajA2omtnlEUOyEKKRXLmH9q0L-PpMF8&authuser=1 draft]) | |||
** bugs with in-testsuite coverage ([https://drive.google.com/open?id=119Cyu-hZWrV0dXhcQ1HMPDIRISZf5R-b9dyKjulIIbk&authuser=1 draft]) | |||
** tracked bugs by version ([https://drive.google.com/open?id=1yw4tQ9g20DlkIltVa1XZjnC9Gptq8lmpJGD1bY4zFlw&authuser=1 draft]) | |||
** unconfirmed bugs by sub-component ([https://drive.google.com/open?id=1DhBKA4P6jEVenV7Z0L043x8MV4vfbCd6qlNPA_8zP4Y&authuser=1 draft]) | |||
** mean time to find regression window | |||
** mean time to resolve a bug with a regression window vs a regression without a window | |||
* ''to be determined'' | |||
** spec test coverage | |||