Auto-tools/Projects/DevelopmentMetrics: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(ordering)
(fix links)
 
(20 intermediate revisions by the same user not shown)
Line 2: Line 2:


These charts serve two purposes.  The first is to satisfy one of the specific needs of the [[Bugzilla Anthropology|Bugzilla Anthropology Project]] which was to provide a view into patch review times.  The second is to prove
These charts serve two purposes.  The first is to satisfy one of the specific needs of the [[Bugzilla Anthropology|Bugzilla Anthropology Project]] which was to provide a view into patch review times.  The second is to prove
ElasticSearch can be used a performant [http://en.wikipedia.org/wiki/Data_warehouse|data warehouse].
ElasticSearch can be used a performant [http://en.wikipedia.org/wiki/Data_warehouse data warehouse].
More on the project and conclusions can be found at the [https://wiki.mozilla.org/index.php?title=Bugzilla_Anthropology/2013-01-29|Jan 2013 project summary].
More on the project and conclusions can be found at the [https://wiki.mozilla.org/index.php?title=Bugzilla_Anthropology/2013-01-29 Jan 2013 project summary].
There is also my [http://people.mozilla.org/~klahnakoski/docs/Bugs%20in%20ES.pdf|DashCon presentation]( [http://people.mozilla.org/~klahnakoski/docs/Bugs%20in%20ES.pptx|PPT])
There is also my [https://github.com/klahnakoski/Bugzilla-ETL/blob/v2/docs/Bugzilla%20Lightning%20161207.pdf DashCon presentation]( [https://github.com/klahnakoski/Bugzilla-ETL/blob/v2/docs/Bugzilla%20Lightning%20161207.ppt PPT])
 


== Bug Counts ==
== Bug Counts ==


{|
{| class="wikitable"
|-
|-
| [http://people.mozilla.org/~klahnakoski/modevmetrics/Bug-Counts.html Bug Counts] || Simple count of open bugs over time, with open and close rates, and median age
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/metrics/Bug-Counts.html Bug Counts] || Open bug count over time, with open/close rates, and median age.  Filter bugs by time range, components, and common program markups.  This dashboard is meant as a rough-first-step to building something more process specific.
chart.
|-
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/metrics/Bug-CloseRate.html#sampleInterval=week&sampleMax=2014-01-25&sampleMin=2012-05-27 Bug Close Rate] || Counts bug closures, and requires writing Elasticsearch filters to limit view to desired category of bugs.
|}
 
==Patch Reviews==
 
<span style="color:red">Reviews are broken [https://bugzilla.mozilla.org/show_bug.cgi?id=1344253 see bug 1344253]</span>
 
{| class="wikitable"
|-
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/review/Review-byTop.html Top 30 Reviewers]<br><span style="color:red">Requires VPN. Allow mixed content.</span>
| Shows the top reviewers from the past 6 weeks, the number of reviews they did, and the number in their queue right now.  The original objective of this dashboard is to confirm some engineers have too many reviews. This dashboard is fine for identifying egregious differences, but poor for individual comparison; it seems personal style, team style, and code under review all impact the size and number of reviews done.
|-
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/review/ReviewIntensity_First.html Time to First Incoming Review]
<br><span style="color:red">Requires VPN. Allow mixed content.</span>
| Counts review requests and review completion rates.  Also breaks down aggregate review age.  The first review has been shown to take the longest, probably due to people synchronization issues.  Subsequent reviews were almost instantaneous, so we exclude them.  This dashboard is not good for fine grained comparison of teams because the nature of the bugs, and the process used, is often too different.  The response time INCLUDES WEEKENDS.
|-
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/review/Review-byReviewer.html Incoming Review by Individual]<br><span style="color:red">Requires VPN. Allow mixed content.</span>
| Top reviewers, but this time with some breakdown by week and component.  Interesting, but not actionable.
|- style="vertical-align:top;"
| [https://charts.mozilla.org/review/ReviewIntensity.html Incoming Reviews]
<br><span style="color:red">Requires VPN. Allow mixed content.</span>
| Statistics for all reviews.  Subsequent reviews of a bug happen much faster than the first.  Maybe the fast review speed is an administrative anomaly; reviews are added after the fact, and marked as reviewed.  Maybe, the requester and reviewer are working closely to remove the remaining nits.
|-
|-
| [http://people.mozilla.org/~klahnakoski/modevmetrics/Bug-CloseRate.html Bug Close Rate] || Much better array of filters available
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/review/Reviews_Pending_18.html Reviews Requested in the Past 18 weeks]
| An attempt to better understand the aggregate response time; while treating "old" reviews (those over 18weeks), as something else entirely.
|}
|}


==Engineering Dashboards==


==Patch Reviews==
Dashboards for use by developers, or our own team


{|
{| class="wikitable"
|-
|-  
| [http://people.mozilla.org/~klahnakoski/review/Review-byTop.html Top Reviewers] || Shows the top 30 reviewers from the past 6 weeks, the number of reviews they did, and the number in their queue right now
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/NeglectedOranges/index.html Neglected Oranges]
|-
| List of current, intermittent test failures, and have seen no bug activity in the past week.
| [http://people.mozilla.org/~klahnakoski/review/Review-byReviewer.html Reviews by Reviewer] || Top reviewers, but this time with some breakdown by week and component
|-  
|-
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/platform/dashboard.html Management Dashboard]
| [http://people.mozilla.org/~klahnakoski/review/ReviewIntensity_First.html ReviewIntensity_First] || Counts review requests and review completion rates.  Also breaks down aggregate review age.  The first review has been shown to take significantly longer
| View of all bugs for a particular team, in priority order, and categorized to help understand scope. This prototype was designed in 2015 for the categories important to the JS team.
|-
| [http://people.mozilla.org/~klahnakoski/review/ReviewIntensity.html ReviewIntensity] || Statistics for all reviews.  Subsequent reviews of a bug happen much faster than the first. It is assumed once the bug has the attention of the reviewer, the review cycle happens quickly
|}
|}


==Release Management Dashboards==


==Project Dashboards and Burndown==
[https://wiki.mozilla.org/Release_Management Release Management] are responsible for quality, security, and stability.
 
{| class="wikitable"
|-
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/platform/releases.html Release Dashboard]
| All outstanding bugs tracked by the Release Management
|-
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/platform-history/release-history.html Release Uplifts]
| Count the number of patches that have been uplifted to the Aurora and Beta branches.
|}


These dashboards are for showing general project state and burndown.
== Contributors ==


{|
{| class="wikitable"
|-
|-  
| [http://people.mozilla.org/~klahnakoski/modevmetrics/Dashboard_byProject.html Project Dashboard] || Lists assigned bugs, patches in review, and patches awaiting landing
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/contributors/dashboard.html# Contributors Dashboard]
|-
| [https://wiki.mozilla.org/EngineeringProductivity Engineering Productivity's] list of contributor bugs that may benefit from some action on our part.
| [http://people.mozilla.org/~klahnakoski/modevmetrics/Predictive_Simple.html#programFilter=B2G+1.3 Burndown Prediction] || Uses average new bug rate, and average close rate to estimate the time it takes
to close all bugs in the given category.  Compares to close rates necessary to meet due date
|-
| [http://people.mozilla.org/~klahnakoski/modevmetrics/Dashboard-FinalBurndown.html Final Burndown Dashboard] || Basic burndown chart. Showing current bugs over time compared to ideal (linear) burndown
|}
|}


==Specific Projects==
==Project Dashboards and Burndown==


In the past there have been some for specific historical charts, and simple trend analysis.<br>
These dashboards are for showing general project state and burndown.


{|
{| class="wikitable"
|-
|-  
| [http://people.mozilla.org/~klahnakoski/modevmetrics/B2G-TriageRate.html FFOS Bug Triage (Nominations)] || The nomination queue can be filtered by product and component. It depends on Mozilla's standard practice using a tracking flag to mark bug one of "?" (triage), "+" (tracked), "-" (not tracked)
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/quantum/blockers.html Quantum Flow tiles]<br>
|-
| All components with Quantum Flow bugs (qf:p1), color-coded by age
| [http://people.mozilla.org/~klahnakoski/modevmetrics/TEF-CloseRate.html TEF vs B2G Fix Rate] || Compare the fix rates for Boot2Gecko product and TEF+ blockers
|-  
|-
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/QuantumFlow2/QuantumFlowBurndown.html Quantum Flow Burndown]<br><span style="color:red">Requires VPN. Allowed mixed content.</span>
| [http://people.mozilla.org/~klahnakoski/modevmetrics/B2G-CloseRate.html B2G Close Rate] || Derived from Bug Close Rate to show four major B2G projects, and only the Resolved=FIXED.
| Quantum Flow bugs (qf:p1) over time, with projected complete date
|-
|-
| [http://people.mozilla.org/~klahnakoski/modevmetrics/Security_Q1_Goal.html Security's 2013 Q1 Goal] || The Security Teams have an ambitious goal to limit the age of high+ security bugs.  This dashboard is meant to help visualize progress toward those goals.
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/metrics/Dashboard_byProject.html Project Dashboard]<br><span style="color:red">Requires VPN. Allowed mixed content.</span>
|-
| Lists assigned bugs, patches in review, and patches awaiting landing
| [http://people.mozilla.org/~klahnakoski/modevmetrics/Security_byPriority.html Security_byPriority] || Count security bugs by priority, over time
|-  
|-
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/metrics/Predictive_Simple.html#programFilter=B2G+1.3 Burndown Prediction]<br><span style="color:red">Requires VPN. Allowed mixed content.</span>
| [http://people.mozilla.org/~klahnakoski/modevmetrics/Security_byRisk.html Security_byRisk.html] || Show risk assessment by component, over time.  It uses security's scoring method to give a number to each of the components
| Uses average new bug rate, and average close rate to estimate the time it takes
to close all bugs in the given category.  Compares to close rates necessary to meet due date
|-  
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/metrics/Dashboard-FinalBurndown.html Final Burndown Dashboard]<br><span style="color:red">Requires VPN. Allowed mixed content.</span>
| Basic burndown chart. Showing current bugs over time compared to ideal (linear) burndown
|}
|}

Latest revision as of 18:04, 6 July 2017

Overview

These charts serve two purposes. The first is to satisfy one of the specific needs of the Bugzilla Anthropology Project which was to provide a view into patch review times. The second is to prove ElasticSearch can be used a performant data warehouse. More on the project and conclusions can be found at the Jan 2013 project summary. There is also my DashCon presentation( PPT)

Bug Counts

Bug Counts Open bug count over time, with open/close rates, and median age. Filter bugs by time range, components, and common program markups. This dashboard is meant as a rough-first-step to building something more process specific.
Bug Close Rate Counts bug closures, and requires writing Elasticsearch filters to limit view to desired category of bugs.

Patch Reviews

Reviews are broken see bug 1344253

Top 30 Reviewers
Requires VPN. Allow mixed content. 		Shows the top reviewers from the past 6 weeks, the number of reviews they did, and the number in their queue right now. The original objective of this dashboard is to confirm some engineers have too many reviews. This dashboard is fine for identifying egregious differences, but poor for individual comparison; it seems personal style, team style, and code under review all impact the size and number of reviews done.
Time to First Incoming Review


Requires VPN. Allow mixed content.

Counts review requests and review completion rates. Also breaks down aggregate review age. The first review has been shown to take the longest, probably due to people synchronization issues. Subsequent reviews were almost instantaneous, so we exclude them. This dashboard is not good for fine grained comparison of teams because the nature of the bugs, and the process used, is often too different. The response time INCLUDES WEEKENDS.
Incoming Review by Individual
Requires VPN. Allow mixed content. 		Top reviewers, but this time with some breakdown by week and component. Interesting, but not actionable.
Incoming Reviews


Requires VPN. Allow mixed content.

Statistics for all reviews. Subsequent reviews of a bug happen much faster than the first. Maybe the fast review speed is an administrative anomaly; reviews are added after the fact, and marked as reviewed. Maybe, the requester and reviewer are working closely to remove the remaining nits.
Reviews Requested in the Past 18 weeks An attempt to better understand the aggregate response time; while treating "old" reviews (those over 18weeks), as something else entirely.

Engineering Dashboards

Dashboards for use by developers, or our own team

Neglected Oranges List of current, intermittent test failures, and have seen no bug activity in the past week.
Management Dashboard View of all bugs for a particular team, in priority order, and categorized to help understand scope. This prototype was designed in 2015 for the categories important to the JS team.

Release Management Dashboards

Release Management are responsible for quality, security, and stability.

Release Dashboard All outstanding bugs tracked by the Release Management
Release Uplifts Count the number of patches that have been uplifted to the Aurora and Beta branches.

Contributors

Contributors Dashboard Engineering Productivity's list of contributor bugs that may benefit from some action on our part.

Project Dashboards and Burndown

These dashboards are for showing general project state and burndown.

Quantum Flow tiles
 All components with Quantum Flow bugs (qf:p1), color-coded by age
Quantum Flow Burndown
Requires VPN. Allowed mixed content. 		Quantum Flow bugs (qf:p1) over time, with projected complete date
Project Dashboard
Requires VPN. Allowed mixed content. 		Lists assigned bugs, patches in review, and patches awaiting landing
Burndown Prediction
Requires VPN. Allowed mixed content. 		Uses average new bug rate, and average close rate to estimate the time it takes

to close all bugs in the given category. Compares to close rates necessary to meet due date

Final Burndown Dashboard
Requires VPN. Allowed mixed content. 		Basic burndown chart. Showing current bugs over time compared to ideal (linear) burndown
Retrieved from "https://wiki.allizom.org/index.php?title=Auto-tools/Projects/DevelopmentMetrics&oldid=1175181"