Metrics and dashboards

Executive Dashboards

Mozilla gathers a variety of information on the useage and health of our products. This data is critical to evaluating the health of our engineering/product ecosystem and user base. The project intent is to create ways to both query and visualize this data.

V1 is the current live dashboard (april 2015). V2 exists to add data sources and a more fully featiured query set. V3 is when this will be handed over to IT for ongoing work.

This is part of the metrics and dashboards effort - see quality dashboard here []

Project Introduction

• Team Mailing list:
• Team IRC Channel:
• Summary of our plans for this year (links back to main page for the area - ex: platform or desktop have roll-up pages with very high level goals)
• More detailed Roadmap, noting that the further out the more lose the targets are
• Dependencies: feature, partner, resource, etc. - put links to other project pages.

Data sources and key metrics (see doc for summary of key metrics being measured and the known data source gaps).

• • Firefox desktop metrics
  • Firefox mobile metrics
  • Tile Metrics
  • Account metrics
  • Data warehouse [data warehouse]

Links to Current info

V1 Current live as of April 2015

• V1 executive dashboards [1]
• V1 user stories [2]

V2

• V2 User stories [3]
• V2 data sources [4]

V3

Roles and Responsibilities

• Dev Manager: Katie Parlante
• Data Scientist: Rebecca Weiss
• TPM: Thomas Huelbert

Meetings & Communications

Meeting	Day of week	Pacific Time	Eastern Time	Central European Time	Vidyo Room	Notes
"Stand-up" Meetings	Wednesday	8:00AM - 8:30AM	11:00AM - 11:30PM	5:00PM - 5:30PM	webRTC-Apps	etherpad
"Pull Meetings"	TBD	8:00AM - 8:30AM	11:00AM - 11:30PM	5:00PM - 5:30PM	webRTC-Apps	etherpad
"Retrospective"	TBD	8:00AM - 9:00AM	11:00AM - 12:00PM	5:00PM - 6:00PM	webRTC-Apps	etherpad

• Links to mailing list or all communication channels

Development

Schedule

Release timing

• Firefox main release schedule is the master location for the dates when each Firefox version goes to Aurora, Beta, & Release.

Sprints (if team is using 2 week sprints)

• schedule for 2 week increments (3 Sprints per Fx Release)
  • There is a field in Bugzilla there is a field for "Iteration" that has the iteration number and the last day in that sprint:
    • <FxRelease#>.1 - <last day before uplift>
    • <FxRelease#>.2 - <last day before uplift>
    • <Fxrelease#>.3 - <last day before uplift>

Upcoming Priorities and Project Health

As we plan what's coming next, these are areas being discussed. This is not a commitment to the next projects - just our scratch area, but it is in order of relative priority. (after the work we've pulled for the sprints).

• finishing Theme 1 (large change)
• Theme 2: any time constraints/dependencies
• Theme 3: quick why

Things we want to do - but parking lot until higher priorities are done:

• Process Improvement: Theme
• Other platform Investigation
• Tech-debt: any time constraints/dependencies
• Theme 4
• Quality: description of areas
• Theme 5

Risks

• Risk 1 : See this doc for known data gaps see data gaps section
• Risk 2 : iOS is not currently in scope but will be released on June 2 (planned). There seems to be a risk of not including this product is the intent is to provide an over arching look at the ecosystem.

Definitions of start and done

Start (its possible not all these items need to be defined or agreed on by all members of the team, but the lack of these items should indicate red flags)

• PRD's have been created: in progress
• User stories created: in progress
• Above requirements are understood (including dependencies such as: access to staging, new hardware, skill set development etc): TBD
• Dev, QE and PM resources are mapped to intended goals: TBD
• schedules create and agreed on: TBD

Done:

• TBD

Current Status

example from Jenn's

Project Health

• (get central location for graphics from Erin)

Include clear, executive level summary that will be included at the [mana page overall view level:

• for company goal x, we are in _____ state because ______. Please consider ______(propose fix or adjustment of goal).
• for release goal for ______ , we are in _____ state because ______. Please consider ______(propose fix or adjustment of goal).

Current (queries)

• Many options for how teams want to present this information.
• Sample queries to visualize the bugs in bugzilla
• Some examples of teams using Iteration, by Release, by tracking bug

Past

• Pipeline of work that has been completed and moving through Aurora and Beta to release.
• Ideally these aren't just queries, but human readable so folks know what is coming in x release.

Product Backlog

Key Bugzilla Queries

• put key queries here so folks can find simply

All work related to the ongoing development and maintenance of the Firefox Desktop Product are collected and prioritized in the Product Backlog. The goals of the Product Backlog are to:

• Enable work to be prioritized so that the team is always working on the most important features.
• Support continual planning as the product emerges so the plan matches reality.
• Improve forecasts so that the stakeholders make the best decisions about the direction of the product.

Bugzilla Agile fields

Several Fields have been added this year into bugzilla for Agile tracking that you may want to leverage:

• Iteration: right now this has the 3 previous and future 2 week sprints for Firefox release schedule - shows iteration (40.1) and uplift date.
• Points: Fibonacci series 1,2,3,5,8,13]
• Rank: see Rank use description under Triage Guidelines
  • Rank will not show under your Product::Component unless you request it by filing a bug with "bugzilla.mozilla.org :: Administration". You need to give a list of acceptable editors (usually the folks that can triage).

Product Backlog

• There needs to be a way to mark the bugs you have triaged versus not. Firefox uses the flag for "firefox-backlog".
• If you are not strictly desktop, there's also an option to add you team to the blocking-flag "backlog".
  • ex: the webRTC team uses webRTC+ for bugs they are accepting and "parking lot" for ones they are not. The queries are based on product/component - so multiple teams can use "parkinglot" and just request a new tag <team name>+ tag by filing a bug under "bugzilla.mozilla.org :: Administration"
• ex: Bugzilla Ranked "Firefox-backlog+" list
  • Add the "Rank" Column to your results and sort on Rank to view order of work.
    • The option to "Change columns" is at bottom of search results
  • Search is based on "Flags" - "contains the string" - firefox-backlog+
• Un-triaged bugs
  • Search is based on not having the Firefox-backlog+ or Firefox-backlog-

Triage Guidelines

The Product Backlog is continually maintained by the Hello Management team to ensure new priorities are available for each Sprint Planning meeting.

• Priorities follow this Standard:
  • Priority 1 - Blocker, must-fix before shipping.
  • Priority 2 - Major impact, considering severity × probability. Not a blocker for shipping.
  • Priority 3 - Average Bug. definitely a problem, but doesn't stop someone from using the product.
  • Priority 4 - Minor or polish bugs that are real issues (especially in aggregate) and annoying.
  • Priority 5 - Low-impact. something we'd fix, but mostly only bothers the discerning user. Little impact on usability.

• RANK: As priority buckets start to have a large amount of bugs in them, the Rank field can be used to call attention to higher or lower rank and provide a way to sort easily in bugzilla. To have some rhyme/reason to the order - Rank should relate to Priority. The "Ranking" number does not need to be unique. Unless there is a reason to for a bug to be considered before (or after) others in the Priority bucket - default to mid-range value.
  • P1 Rank options=1-19, default 15
  • P2 Rank options=20-29, default 25
  • P3 Rank options=30-39, default 35
  • P4 Rank options=40-49, default 45
  • P5 Rank options=50-59, default 55
  • any that we don't think we can get to in the next 6 months should go in "backlog-" area

• The Firefox-backlog flag is used to track bugs that are approved for the Backlog "+"
• QE-Verify is a flag that developers should be setting on bugs they are working on. This is used for QE to filter which bugs they check
  • "+" means that QE should look at the bug and it can be verified with human eyes
  • "-" means QE should not look at
    • Typically goes with in-testsuite set to "+", to show testing via another method.
• "Points" should be set when known (if nothing set - assumed a "1" or very small). Most relevant if taking a bigger bug so we know when bugs are large bits of work.
• "Iteration" should be updated when a bug is being worked on during a particular Sprint.

Filing a bug

• • Open a bug under Product:"Loop" || Component: "General" or "Client"
  • Hello team reviews for inclusion into a release backlog every 2 weeks, and will mark "firefox-blocking" accordingly
• If there is a bug that should be considered for taking ASAP - you can mark "firefox-backlog"+
  • Before it can be given a Rank it should:
    • be in an actionable state
    • for defects, the problem is ready for Engineering or UX: diagnosis, measurement, design, or fixing
    • for feature requests or enhancements, it means that there's a clear problem statement or suggestion
    • has a difficulty/user-impact ratio low enough that we can reasonably expect to spend time fixing the bug within the next 6 months

Templates

Go to Templates page for sample etherpads, invite details, etc. for the following types of meetings.

• stand-ups (daily, weekly, or other)
• retrospective invites with questions/structure of the meeting in etherpad
• cross-team/company meeting invites/etherpads to keep efficient and only when needed
• postmortems

Archive

as iteration information or links get old or if this is replacing an older page and we don't want to lose path to the information...link to an archive page from here.

Reference materials

Sample bugzilla Queries

Please add your queries and descriptions of what they are doing here to help other create without having to learn bugzilla!

• Queries to pull info for an iteration

Tools

• Aha
• Trello

https://waffle.io/mozilla/payments

Reading

• Agile Retrospectives: Making Good Teams Great by Esther Derby
• https://pragprog.com/book/dlret/agile-retrospectives
• http://www.jamesshore.com/Agile-Book/retrospectives.html
• https://www.crisp.se/file-uploads/Lean-from-the-trenches.pdf
• http://shop.oreilly.com/product/0636920033851.do user story mapping as a too
• Buidling great teams https://hbr.org/2012/04/the-new-science-of-building-great-teams