QA/Platform/DOM/Metrics: Difference between revisions

From MozillaWiki
< QA‎ | Platform‎ | DOM
Jump to navigation Jump to search
 
(9 intermediate revisions by the same user not shown)
Line 12: Line 12:
In implementation we will experiment with a lot of different metrics and data sources in an attempt to see which metrics, either singularly or in conjunction with other data, enable us to effectively make informed decisions.
In implementation we will experiment with a lot of different metrics and data sources in an attempt to see which metrics, either singularly or in conjunction with other data, enable us to effectively make informed decisions.


=== Stability ===
=== Drafted ===
* [https://docs.google.com/a/mozilla.com/spreadsheets/d/1T0ZMPtojYsWcd1oNmCVypV9cJgnnx-VYcDv7P0t945c/pubchart?oid=1017897746&format=interactive Crash bugs reported/resolved month to month for 2014]
The following metrics were developed using data from [http://people.mozilla.org/~klahnakoski/modevmetrics/ Kyle Lahnakoski's] elastic search Bugzilla database. Pending peer review, the next step will be to evaluate [https://github.com/mozilla/metrics-graphics better charting] and eventually a more automated solution.
* [https://docs.google.com/a/mozilla.com/spreadsheets/d/1T0ZMPtojYsWcd1oNmCVypV9cJgnnx-VYcDv7P0t945c/pubchart?oid=932225928&format=interactive Crash bugs reported/resolved month to month for 2014 in aggregate] <sup>1</sup>
* [https://docs.google.com/a/mozilla.com/spreadsheets/d/1i9jGwCZDOXJlSGrmIgvyHzBQbTG5aPEi6Np_P-4GDwk/edit?usp=sharing Unresolved crash bugs by component] - ''possibly correlate to KairRo's reports:''
* [https://docs.google.com/a/mozilla.com/spreadsheets/d/1T0ZMPtojYsWcd1oNmCVypV9cJgnnx-VYcDv7P0t945c/pubchart?oid=172873160&format=interactive Crash bugs reported month to month by component for 2014] <sup>1</sup>
** https://crash-analysis.mozilla.com/rkaiser/2015-02-05/2015-02-05.firefox.release.35.0.components.html
* [https://docs.google.com/a/mozilla.com/spreadsheets/d/1T0ZMPtojYsWcd1oNmCVypV9cJgnnx-VYcDv7P0t945c/pubchart?oid=763311749&format=interactive Crash bugs reported month to month by component for 2014 in aggregate] <sup>1</sup>
** https://crash-analysis.mozilla.com/rkaiser/2015-02-05/2015-02-05.firefox.beta.36.0.components.html
* [https://docs.google.com/a/mozilla.com/spreadsheets/d/1i9jGwCZDOXJlSGrmIgvyHzBQbTG5aPEi6Np_P-4GDwk/pubchart?oid=1967662858&format=interactive Crash bugs unresolved week to week by component since January 1, 2013] <sup>2</sup>
** https://crash-analysis.mozilla.com/rkaiser/2015-02-05/2015-02-05.firefox.aurora.components.html
* [https://docs.google.com/a/mozilla.com/spreadsheets/d/1i9jGwCZDOXJlSGrmIgvyHzBQbTG5aPEi6Np_P-4GDwk/pubchart?oid=1323458719&format=interactive Crash bugs reported/resolved week to week since January 1, 2013] <sup>2</sup>
** https://crash-analysis.mozilla.com/rkaiser/2015-02-05/2015-02-05.firefox.nightly.components.html
* [https://docs.google.com/a/mozilla.com/spreadsheets/d/1i9jGwCZDOXJlSGrmIgvyHzBQbTG5aPEi6Np_P-4GDwk/pubchart?oid=534229400&format=interactive Crash bugs reported/resolved week to week since January 1, 2013 in aggregate] <sup>2</sup>
* <strike>[https://docs.google.com/a/mozilla.com/spreadsheets/d/1epRHLg58XmW90-SwCzFmmhqy-_SAeoh76_F9WXi-NQc/edit?usp=sharing Unresolved intermittent failure bugs by component]</strike> ''Deprioritized for further investigation following discussion with Andrew Overholt''
 
* [https://docs.google.com/a/mozilla.com/spreadsheets/d/1lJSH_pnnj0k-fYP_W4jZ9GYuEe7DW32YwuL1nw61rd8/edit?usp=sharing Unresolved regression bugs by component]
<sup>1</sup> ''generated using manual bugzilla queries''<br>
* [https://docs.google.com/a/mozilla.com/spreadsheets/d/18JHF8otni_5bJHFVqt6ImsVuX5MQrSLW23fXX0QOAQc/edit?usp=sharing Unconfirmed bugs by component]
<sup>2</sup> ''generated using Kyle Lahnakoski's elastic search bugzilla database''
* [https://github.com/ashughes1/mozbackouts Backouts by component]
* [https://docs.google.com/a/mozilla.com/spreadsheets/d/1pafr15_q0ztifEMY-bHEQWlQBkPrmq7e3ebU8JS_vqg/edit#gid=1448481380 Regressions resolved prior to Beta] (updated as of Firefox 37)


=== Pending Investigation ===
=== Pending Investigation ===
Line 57: Line 58:
** FHR
** FHR
** Socorro
** Socorro
** [http://release.mozilla.org/statistics/37/2015/03/13/fx-37-b4-to-b5.html Release Management]
* [http://brasstacks.mozilla.com/orangefactor/ OrangeFactor]


== Bugzilla Data ==
== Bugzilla Data ==
=== Using Manual Querying ===
=== Using Manual Querying ===
=== Using Elastic Search ===
=== Using Elastic Search ===

Latest revision as of 15:21, 16 May 2015

This page documents the various metrics QA uses to assess the quality of Firefox/Gecko's DOM implementation from version to version. The purpose of these metrics is to help us make informed decisions about testing strategy. These metrics serve several needs, including but not limited to:

  • identifying areas of risk for testing before we release a new version of Firefox
  • informing root cause analysis discussions when something inevitably slips through the cracks
  • measuring the trends in product quality over the long-term
  • measuring competitiveness and spec compliance

Please contact Anthony Hughes if you have any questions or feedback regarding this initiative. He can usually be found in #qa on irc.mozilla.org between 09:00 and 17:00 (UTC-9/-10) Monday to Friday by mentioning the nickname ashughes.

Phase I: Establishing a Baseline

The goal of the initial phase is to establish a baseline that all future quality can be measured against.

In implementation we will experiment with a lot of different metrics and data sources in an attempt to see which metrics, either singularly or in conjunction with other data, enable us to effectively make informed decisions.

Drafted

The following metrics were developed using data from Kyle Lahnakoski's elastic search Bugzilla database. Pending peer review, the next step will be to evaluate better charting and eventually a more automated solution.

Pending Investigation

Draft proof of concepts can be found in my Google Drive

Bugzilla Data

Using Manual Querying

Using Elastic Search

Retrieved from "https://wiki.allizom.org/index.php?title=QA/Platform/DOM/Metrics&oldid=1074921"