Connected Devices/Projects/Metrics
With ideas becoming more mature during the innovation process - and eventually becoming products - monitoring progress, measuring market fit or understanding the customer lifecycle becomes more and more important for Product teams. This is also known as Innovation Accounting. Even early in the innovation and product development process, testing hypotheses through experiments and creating insights from the data collected is an essential part of the Lean Startup approach.
For all of the Connected Devices products, collecting metrics is essential to guide decision making during the gating process. One problem with the set of Connected Devices products is that these products are going to be very different in terms of functionalities and technologies. Unlike FirefoxOS where one set of metrics was good for all supported devices, we will need very specific solutions for each product that will ideally follow a common structure of Innovation Accounting.
In addition, accessing data from different products should require as low effort as possible for decision makers in Connected Devices (engineers, PMs, senior management…).
Project Overview
Our goal is to build a framework that will allow each train to collect the specific data needed, but still use the unified Telemetry Pipeline to send data to the metrics infrastructure. So we will be able to rely on architecture that already exists in Mozilla and which has running processes for validating, sanitizing and even automatically analyzing device data.
Having the Telemetry API as a common bearer is not enough as we need to ensure the information sent is consistent. For doing so, we will rely on the Telemetry "Ping" mechanism that is already used in many Mozilla products.
In order to ensure information is consistent, can be easily processed and aggregated, we have defined one common Connected Devices Ping Format proposal that could be implemented by each train at the gate 1+ phase.
Connected Devices will be aiming to use a common ping format across all of Connected Devices. In summary, this currently consists of three ping formats:
- FTU Ping - Typically sent after the device is started for the first time. This helps to determine how many Active devices there are.
- Core Ping - Contains histograms that developers have instrumented in their code using the Metrics Library. Sent in a recurring interval such as once per day or once every two weeks depending on requirements. Whether or not this is sent is dependent on privacy choice of the user.
- Crash Ping - Sent when there is a crash. Also dependent on privacy choice of user.
These pings will all be submitted to the Telemetry data pipeline at incoming.telemetry.mozilla.org.Having a common ping format still leaves freedom to CD trains to send specific payloads within the pings as every product will have different aspects to be measured.
Next Steps
- Share the basic ping format with all the different CD trains and get their feedback so we can iterate over it and adapt to the potential needs of any train.
- Build the common CD framework (Controller and Histograms libraries) that can be used by any of the trains independently of the technology chosen for each of them
- Understand where are the trains in terms of measurement and which kind of metrics they consider important.
- Test the proposal with one train to be used as benchmark
- Help all the teams with Telemetry API adoption, Data Analysis and Reporting.
Program Status
| Milestone | Date | Status | Comments |
|---|---|---|---|
| Connected Devices Metrics Library Architecture Design | 4/08/2016 | ON TARGET | |
| Presentation for Connected Devices Product Management | End of April | ON TARGET | Product Management meetup has been delayed, no final date yet but the earlier date would end of April. |
| CD Metrics Milestones | NOT STARTED | Metrics milestones will be added as milestones once they're identified | |
| London All Hands Conference Presentation | 6/13/2016 | NOT STARTED |
Status Key
| Color | Status | Key |
|---|---|---|
| On Target | The project or deliverable is expected to meet its due date. | |
| Challenged | The project or deliverable is facing an issue that might cause it to miss its due date, but a “get well” plan has been developed to get it back on track. | |
| At Risk or Late | The project or deliverable is blocked or facing an issue that might cause it to miss its due date, and there’s no “get well” plan to get it back on track, or it is already late. | |
| Done | The project or deliverable has been completed. | |
| On Hold or Not Started | The project or deliverable has either not been started or has been placed on hold. |
Archived Completed Deliverables
| Milestone | Date | Status | Status Notes |
|---|---|---|---|
| Connected Devices Ping Format | 3/04/2016 | Done | CD Common Ping format |
| Connected Devices Crash Ping Format drafted | 3/11/2016 | Done | Connected Connected Devices Ping Format proposal |
| High level Histograms Architecture Design | 4/08/2016 | Done | Histograms Architecture Design Diagram |
| Connected Devices Dashboard and Analysis Requirements | 4/08/2016 | On target | Connected Devices Dashboard and Analysis Requirements |
Program Timeline
This timeline is simply a placeholder. Timeline to be inserted once milestones and schedule are known.
Project Management
We use Trello board for project management. All ongoing tasks are listed there.
Sprint Backlog
Development process
Repositories
- Yoric's telemetry library: https://github.com/Yoric/telemetry.rs
- Metrics controller: https://github.com/tamarahills/metrics_controller
IRC
You can find us on irc.mozilla.org in the #cd-metrics channel.
Methodology
The project is in early state but we have planned to use some agile practices:
- Weekly meetings in CD_Metrics Vidyo room on Wednesday at 5:15 PM UTC (9:15 AM, PST), notes in Google doc
- Sprint Planning meeting every other Monday in CD_Metrics Vidyo room at 5:15 PM UTC (9:15 AM, PST)
- Daily standup meetings everyday except the days with Sprint Planning or Weekly meeting in CD_Metrics Vidyo room at 5:15 PM UTC (9:15 AM, PST). You can include your daily report in this etherpad in case you cannot attend the meeting
Calendar
The CD Metrics team has a public calendar with every team meeting Calendar
Instructions for Adding to your Calendar
- Open the calendar.
- Click on the "+ Google Calendar" button in the very bottom right of your screen.
You can also use the XML and ICS methods, but these are not recommended.
Note: The "Find a Time" feature will not work for other people if you import this calendar. As a consequence, others will not see that you are unavailable when attending a FlyWeb meeting. We suggest either accepting this, or adding the meetings to your main calendar as well.
References
Team
- Dylan Oliver - Engineering Manager
- Tamara Hills - Engineering
- Russ Nicoletti - Engineering
- Michael Bryant - QA/Engineering
- Maria Oteo - EPM