Confirmed users
14,525
edits
QA/Platform/DOM/Metrics: Difference between revisions
→Pending Investigation
| (19 intermediate revisions by the same user not shown) | |||
| Line 12: | Line 12: | ||
In implementation we will experiment with a lot of different metrics and data sources in an attempt to see which metrics, either singularly or in conjunction with other data, enable us to effectively make informed decisions. | In implementation we will experiment with a lot of different metrics and data sources in an attempt to see which metrics, either singularly or in conjunction with other data, enable us to effectively make informed decisions. | ||
=== | === Drafted === | ||
The following metrics were developed using data from [http://people.mozilla.org/~klahnakoski/modevmetrics/ Kyle Lahnakoski's] elastic search Bugzilla database. Pending peer review, the next step will be to evaluate [https://github.com/mozilla/metrics-graphics better charting] and eventually a more automated solution. | |||
* [https://docs.google.com/a/mozilla.com/spreadsheets/d/1i9jGwCZDOXJlSGrmIgvyHzBQbTG5aPEi6Np_P-4GDwk/edit?usp=sharing Unresolved crash bugs by component] - ''possibly correlate to KairRo's reports:'' | |||
** https://crash-analysis.mozilla.com/rkaiser/2015-02-05/2015-02-05.firefox.release.35.0.components.html | |||
** https://crash-analysis.mozilla.com/rkaiser/2015-02-05/2015-02-05.firefox.beta.36.0.components.html | |||
** https://crash-analysis.mozilla.com/rkaiser/2015-02-05/2015-02-05.firefox.aurora.components.html | |||
** https://crash-analysis.mozilla.com/rkaiser/2015-02-05/2015-02-05.firefox.nightly.components.html | |||
* <strike>[https://docs.google.com/a/mozilla.com/spreadsheets/d/1epRHLg58XmW90-SwCzFmmhqy-_SAeoh76_F9WXi-NQc/edit?usp=sharing Unresolved intermittent failure bugs by component]</strike> ''Deprioritized for further investigation following discussion with Andrew Overholt'' | |||
* [https://docs.google.com/a/mozilla.com/spreadsheets/d/1lJSH_pnnj0k-fYP_W4jZ9GYuEe7DW32YwuL1nw61rd8/edit?usp=sharing Unresolved regression bugs by component] | |||
* [https://docs.google.com/a/mozilla.com/spreadsheets/d/18JHF8otni_5bJHFVqt6ImsVuX5MQrSLW23fXX0QOAQc/edit?usp=sharing Unconfirmed bugs by component] | |||
* [https://github.com/ashughes1/mozbackouts Backouts by component] | |||
* [https://docs.google.com/a/mozilla.com/spreadsheets/d/1pafr15_q0ztifEMY-bHEQWlQBkPrmq7e3ebU8JS_vqg/edit#gid=1448481380 Regressions resolved prior to Beta] (updated as of Firefox 37) | |||
=== Pending Investigation === | |||
''Draft proof of concepts can be found in my [https://drive.google.com/drive/u/1/#folders/0B9_ZEZ4TIS-8M2ZaTDFXT20wMDA Google Drive]'' | ''Draft proof of concepts can be found in my [https://drive.google.com/drive/u/1/#folders/0B9_ZEZ4TIS-8M2ZaTDFXT20wMDA Google Drive]'' | ||
* https://github.com/AutomatedTester/futurama-data | * https://github.com/AutomatedTester/futurama-data | ||
** backout rate | ** backout rate | ||
* http://brasstacks.mozilla.com/testreports/weekly | * http://brasstacks.mozilla.com/testreports/weekly | ||
| Line 30: | Line 42: | ||
** regression bugs by number of comments ([https://drive.google.com/open?id=1ZZL7WU3H5h9sM_1XVJ1HF7oD-s4q3EwSy-sGu9NLeOg&authuser=1 draft]) | ** regression bugs by number of comments ([https://drive.google.com/open?id=1ZZL7WU3H5h9sM_1XVJ1HF7oD-s4q3EwSy-sGu9NLeOg&authuser=1 draft]) | ||
** regression bugs in/out ([https://drive.google.com/open?id=1NzjmQP7Io2rWajA2omtnlEUOyEKKRXLmH9q0L-PpMF8&authuser=1 draft]) | ** regression bugs in/out ([https://drive.google.com/open?id=1NzjmQP7Io2rWajA2omtnlEUOyEKKRXLmH9q0L-PpMF8&authuser=1 draft]) | ||
** regression bugs by sub-component ([https://drive.google.com/open?id=1WNfBIk15p0KQ-Ha__jI74u46qfcMbht6cgG54UQDoDw&authuser=1 draft]) | |||
** bugs with in-testsuite coverage ([https://drive.google.com/open?id=119Cyu-hZWrV0dXhcQ1HMPDIRISZf5R-b9dyKjulIIbk&authuser=1 draft]) | ** bugs with in-testsuite coverage ([https://drive.google.com/open?id=119Cyu-hZWrV0dXhcQ1HMPDIRISZf5R-b9dyKjulIIbk&authuser=1 draft]) | ||
** tracked bugs by version ([https://drive.google.com/open?id=1yw4tQ9g20DlkIltVa1XZjnC9Gptq8lmpJGD1bY4zFlw&authuser=1 draft]) | ** tracked bugs by version ([https://drive.google.com/open?id=1yw4tQ9g20DlkIltVa1XZjnC9Gptq8lmpJGD1bY4zFlw&authuser=1 draft]) | ||
** unconfirmed bugs by sub-component ([https://drive.google.com/open?id=1DhBKA4P6jEVenV7Z0L043x8MV4vfbCd6qlNPA_8zP4Y&authuser=1 draft]) | ** unconfirmed bugs by sub-component ([https://drive.google.com/open?id=1DhBKA4P6jEVenV7Z0L043x8MV4vfbCd6qlNPA_8zP4Y&authuser=1 draft]) | ||
** bugs by platform ([https://drive.google.com/open?id= | ** bugs by platform ([https://drive.google.com/open?id=1tU0decofkJS0fcklcHIlmzZJJy1FYYfOyhsXsoxa9Zs&authuser=1 draft]) | ||
** mean time to find regression window | ** mean time to find regression window [ask klahnakoski about elastic search db] | ||
** mean time to resolve a bug with a regression window vs a regression without a window | ** mean time to resolve a bug with a regression window vs a regression without a window [ask klahnakoski about elastic search db] | ||
* ''to be determined'' | * ''to be determined'' | ||
** spec test coverage | ** spec test coverage | ||
| Line 41: | Line 54: | ||
** https://treeherder.mozilla.org/#/jobs?repo=mozilla-central | ** https://treeherder.mozilla.org/#/jobs?repo=mozilla-central | ||
** https://scitools.com/ for code complexity analysis, possibly other metrics | ** https://scitools.com/ for code complexity analysis, possibly other metrics | ||
** http://metricsgraphicsjs.org/ for JS library to create dashboards | |||
** Telemetry | |||
** FHR | |||
** Socorro | |||
** [http://release.mozilla.org/statistics/37/2015/03/13/fx-37-b4-to-b5.html Release Management] | |||
* [http://brasstacks.mozilla.com/orangefactor/ OrangeFactor] | |||
== Bugzilla Data == | |||
=== Using Manual Querying === | |||
=== Using Elastic Search === | |||