Auto-tools/Projects/DevelopmentMetrics: Difference between revisions
< Auto-tools | Projects
Klahnakoski (talk | contribs) (remove vpn indicator) |
Klahnakoski (talk | contribs) (fix links) |
||
| (5 intermediate revisions by the same user not shown) | |||
| Line 4: | Line 4: | ||
ElasticSearch can be used a performant [http://en.wikipedia.org/wiki/Data_warehouse data warehouse]. | ElasticSearch can be used a performant [http://en.wikipedia.org/wiki/Data_warehouse data warehouse]. | ||
More on the project and conclusions can be found at the [https://wiki.mozilla.org/index.php?title=Bugzilla_Anthropology/2013-01-29 Jan 2013 project summary]. | More on the project and conclusions can be found at the [https://wiki.mozilla.org/index.php?title=Bugzilla_Anthropology/2013-01-29 Jan 2013 project summary]. | ||
There is also my [ | There is also my [https://github.com/klahnakoski/Bugzilla-ETL/blob/v2/docs/Bugzilla%20Lightning%20161207.pdf DashCon presentation]( [https://github.com/klahnakoski/Bugzilla-ETL/blob/v2/docs/Bugzilla%20Lightning%20161207.ppt PPT]) | ||
== Bug Counts == | == Bug Counts == | ||
| Line 10: | Line 10: | ||
{| class="wikitable" | {| class="wikitable" | ||
|- | |- | ||
| style="vertical-align:top;width:300px;" | [ | | style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/metrics/Bug-Counts.html Bug Counts] || Open bug count over time, with open/close rates, and median age. Filter bugs by time range, components, and common program markups. This dashboard is meant as a rough-first-step to building something more process specific. | ||
|- | |- | ||
| style="vertical-align:top;width:300px;" | [ | | style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/metrics/Bug-CloseRate.html#sampleInterval=week&sampleMax=2014-01-25&sampleMin=2012-05-27 Bug Close Rate] || Counts bug closures, and requires writing Elasticsearch filters to limit view to desired category of bugs. | ||
|} | |} | ||
==Patch Reviews== | ==Patch Reviews== | ||
<span style="color:red">Reviews are broken [https://bugzilla.mozilla.org/show_bug.cgi?id=1344253 see bug 1344253]</span> | |||
{| class="wikitable" | {| class="wikitable" | ||
|- | |- | ||
| style="vertical-align:top;width:300px;" | [ | | style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/review/Review-byTop.html Top 30 Reviewers]<br><span style="color:red">Requires VPN. Allow mixed content.</span> | ||
| Shows the top reviewers from the past 6 weeks, the number of reviews they did, and the number in their queue right now. The original objective of this dashboard is to confirm some engineers have too many reviews. This dashboard is fine for identifying egregious differences, but poor for individual comparison; it seems personal style, team style, and code under review all impact the size and number of reviews done. | | Shows the top reviewers from the past 6 weeks, the number of reviews they did, and the number in their queue right now. The original objective of this dashboard is to confirm some engineers have too many reviews. This dashboard is fine for identifying egregious differences, but poor for individual comparison; it seems personal style, team style, and code under review all impact the size and number of reviews done. | ||
|- | |- | ||
| style="vertical-align:top;width:300px;" | [ | | style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/review/ReviewIntensity_First.html Time to First Incoming Review] | ||
<br><span style="color:red">Requires VPN. Allow mixed content.</span> | |||
| Counts review requests and review completion rates. Also breaks down aggregate review age. The first review has been shown to take the longest, probably due to people synchronization issues. Subsequent reviews were almost instantaneous, so we exclude them. This dashboard is not good for fine grained comparison of teams because the nature of the bugs, and the process used, is often too different. The response time INCLUDES WEEKENDS. | | Counts review requests and review completion rates. Also breaks down aggregate review age. The first review has been shown to take the longest, probably due to people synchronization issues. Subsequent reviews were almost instantaneous, so we exclude them. This dashboard is not good for fine grained comparison of teams because the nature of the bugs, and the process used, is often too different. The response time INCLUDES WEEKENDS. | ||
|- | |- | ||
| style="vertical-align:top;width:300px;" | [ | | style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/review/Review-byReviewer.html Incoming Review by Individual]<br><span style="color:red">Requires VPN. Allow mixed content.</span> | ||
| Top reviewers, but this time with some breakdown by week and component. Interesting, but not actionable. | | Top reviewers, but this time with some breakdown by week and component. Interesting, but not actionable. | ||
|- style="vertical-align:top;" | |- style="vertical-align:top;" | ||
| [ | | [https://charts.mozilla.org/review/ReviewIntensity.html Incoming Reviews] | ||
<br><span style="color:red">Requires VPN. Allow mixed content.</span> | |||
| Statistics for all reviews. Subsequent reviews of a bug happen much faster than the first. Maybe the fast review speed is an administrative anomaly; reviews are added after the fact, and marked as reviewed. Maybe, the requester and reviewer are working closely to remove the remaining nits. | | Statistics for all reviews. Subsequent reviews of a bug happen much faster than the first. Maybe the fast review speed is an administrative anomaly; reviews are added after the fact, and marked as reviewed. Maybe, the requester and reviewer are working closely to remove the remaining nits. | ||
|- | |- | ||
| style="vertical-align:top;width:300px;" | [ | | style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/review/Reviews_Pending_18.html Reviews Requested in the Past 18 weeks] | ||
| An attempt to better understand the aggregate response time; while treating "old" reviews (those over 18weeks), as something else entirely. | | An attempt to better understand the aggregate response time; while treating "old" reviews (those over 18weeks), as something else entirely. | ||
|} | |} | ||
| Line 41: | Line 45: | ||
{| class="wikitable" | {| class="wikitable" | ||
|- | |- | ||
| style="vertical-align:top;width:300px;" | [https:// | | style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/NeglectedOranges/index.html Neglected Oranges] | ||
| List of current, intermittent test failures, and have | | List of current, intermittent test failures, and have seen no bug activity in the past week. | ||
|- | |||
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/platform/dashboard.html Management Dashboard] | |||
| View of all bugs for a particular team, in priority order, and categorized to help understand scope. This prototype was designed in 2015 for the categories important to the JS team. | |||
|} | |} | ||
| Line 51: | Line 58: | ||
{| class="wikitable" | {| class="wikitable" | ||
|- | |- | ||
| style="vertical-align:top;width:300px;" | [ | | style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/platform/releases.html Release Dashboard] | ||
| All outstanding bugs tracked by the Release Management | | All outstanding bugs tracked by the Release Management | ||
|- | |- | ||
| style="vertical-align:top;width:300px;" | [ | | style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/platform-history/release-history.html Release Uplifts] | ||
| Count the number of patches that have been uplifted to the Aurora and Beta branches. | | Count the number of patches that have been uplifted to the Aurora and Beta branches. | ||
|} | |} | ||
| Line 62: | Line 69: | ||
{| class="wikitable" | {| class="wikitable" | ||
|- | |- | ||
| style="vertical-align:top;width:300px;" | [ | | style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/contributors/dashboard.html# Contributors Dashboard] | ||
| [https://wiki.mozilla.org/EngineeringProductivity Engineering Productivity's] list of contributor bugs that may benefit from some action on our part. | | [https://wiki.mozilla.org/EngineeringProductivity Engineering Productivity's] list of contributor bugs that may benefit from some action on our part. | ||
|} | |} | ||
| Line 72: | Line 79: | ||
{| class="wikitable" | {| class="wikitable" | ||
|- | |- | ||
| style="vertical-align:top;width:300px;" | [ | | style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/quantum/blockers.html Quantum Flow tiles]<br> | ||
| All components with Quantum Flow bugs (qf:p1), color-coded by age | |||
|- | |||
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/QuantumFlow2/QuantumFlowBurndown.html Quantum Flow Burndown]<br><span style="color:red">Requires VPN. Allowed mixed content.</span> | |||
| Quantum Flow bugs (qf:p1) over time, with projected complete date | |||
|- | |||
| style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/metrics/Dashboard_byProject.html Project Dashboard]<br><span style="color:red">Requires VPN. Allowed mixed content.</span> | |||
| Lists assigned bugs, patches in review, and patches awaiting landing | | Lists assigned bugs, patches in review, and patches awaiting landing | ||
|- | |- | ||
| style="vertical-align:top;width:300px;" | [ | | style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/metrics/Predictive_Simple.html#programFilter=B2G+1.3 Burndown Prediction]<br><span style="color:red">Requires VPN. Allowed mixed content.</span> | ||
| Uses average new bug rate, and average close rate to estimate the time it takes | | Uses average new bug rate, and average close rate to estimate the time it takes | ||
to close all bugs in the given category. Compares to close rates necessary to meet due date | to close all bugs in the given category. Compares to close rates necessary to meet due date | ||
|- | |- | ||
| style="vertical-align:top;width:300px;" | [ | | style="vertical-align:top;width:300px;" | [https://charts.mozilla.org/metrics/Dashboard-FinalBurndown.html Final Burndown Dashboard]<br><span style="color:red">Requires VPN. Allowed mixed content.</span> | ||
| Basic burndown chart. Showing current bugs over time compared to ideal (linear) burndown | | Basic burndown chart. Showing current bugs over time compared to ideal (linear) burndown | ||
|} | |} | ||
Latest revision as of 18:04, 6 July 2017
Overview
These charts serve two purposes. The first is to satisfy one of the specific needs of the Bugzilla Anthropology Project which was to provide a view into patch review times. The second is to prove ElasticSearch can be used a performant data warehouse. More on the project and conclusions can be found at the Jan 2013 project summary. There is also my DashCon presentation( PPT)
Bug Counts
| Bug Counts | Open bug count over time, with open/close rates, and median age. Filter bugs by time range, components, and common program markups. This dashboard is meant as a rough-first-step to building something more process specific. |
| Bug Close Rate | Counts bug closures, and requires writing Elasticsearch filters to limit view to desired category of bugs. |
Patch Reviews
Reviews are broken see bug 1344253
| Top 30 Reviewers Requires VPN. Allow mixed content. |
Shows the top reviewers from the past 6 weeks, the number of reviews they did, and the number in their queue right now. The original objective of this dashboard is to confirm some engineers have too many reviews. This dashboard is fine for identifying egregious differences, but poor for individual comparison; it seems personal style, team style, and code under review all impact the size and number of reviews done. |
| Time to First Incoming Review
|
Counts review requests and review completion rates. Also breaks down aggregate review age. The first review has been shown to take the longest, probably due to people synchronization issues. Subsequent reviews were almost instantaneous, so we exclude them. This dashboard is not good for fine grained comparison of teams because the nature of the bugs, and the process used, is often too different. The response time INCLUDES WEEKENDS. |
| Incoming Review by Individual Requires VPN. Allow mixed content. |
Top reviewers, but this time with some breakdown by week and component. Interesting, but not actionable. |
| Incoming Reviews
|
Statistics for all reviews. Subsequent reviews of a bug happen much faster than the first. Maybe the fast review speed is an administrative anomaly; reviews are added after the fact, and marked as reviewed. Maybe, the requester and reviewer are working closely to remove the remaining nits. |
| Reviews Requested in the Past 18 weeks | An attempt to better understand the aggregate response time; while treating "old" reviews (those over 18weeks), as something else entirely. |
Engineering Dashboards
Dashboards for use by developers, or our own team
| Neglected Oranges | List of current, intermittent test failures, and have seen no bug activity in the past week. |
| Management Dashboard | View of all bugs for a particular team, in priority order, and categorized to help understand scope. This prototype was designed in 2015 for the categories important to the JS team. |
Release Management Dashboards
Release Management are responsible for quality, security, and stability.
| Release Dashboard | All outstanding bugs tracked by the Release Management |
| Release Uplifts | Count the number of patches that have been uplifted to the Aurora and Beta branches. |
Contributors
| Contributors Dashboard | Engineering Productivity's list of contributor bugs that may benefit from some action on our part. |
Project Dashboards and Burndown
These dashboards are for showing general project state and burndown.
| Quantum Flow tiles |
All components with Quantum Flow bugs (qf:p1), color-coded by age |
| Quantum Flow Burndown Requires VPN. Allowed mixed content. |
Quantum Flow bugs (qf:p1) over time, with projected complete date |
| Project Dashboard Requires VPN. Allowed mixed content. |
Lists assigned bugs, patches in review, and patches awaiting landing |
| Burndown Prediction Requires VPN. Allowed mixed content. |
Uses average new bug rate, and average close rate to estimate the time it takes
to close all bugs in the given category. Compares to close rates necessary to meet due date |
| Final Burndown Dashboard Requires VPN. Allowed mixed content. |
Basic burndown chart. Showing current bugs over time compared to ideal (linear) burndown |