Connected Devices/Projects/Metrics
With ideas becoming more mature during the innovation process - and eventually becoming products - monitoring progress, measuring market fit or understanding the customer lifecycle becomes more and more important for Product teams. This is also known as Innovation Accounting. Even early in the innovation and product development process, testing hypotheses through experiments and creating insights from the data collected is an essential part of the Lean Startup approach.
For all of the Connected Devices products, collecting metrics is essential to guide decision making during the gating process. One problem with the set of Connected Devices products is that these products are going to be very different in terms of functionalities and technologies. Unlike FirefoxOS where one set of metrics was good for all supported devices, we will need very specific solutions for each product that will ideally follow a common structure of Innovation Accounting.
In addition, accessing data from different products should require as low effort as possible for decision makers in Connected Devices (engineers, PMs, senior management…).
Project Overview
Our goal is to build a framework that will allow each train to collect the specific data needed, but still use the unified Telemetry Pipeline to send data to the metrics infrastructure. So we will be able to rely on architecture that already exists in Mozilla and which has running processes for validating, sanitizing and even automatically analyzing device data.
Having the Telemetry API as a common bearer is not enough as we need to ensure the information sent is consistent. For doing so, we will rely on the Telemetry "Ping" mechanism that is already used in many Mozilla products.
In order to ensure information is consistent, can be easily processed and aggregated, we have defined one common Connected Devices Ping Format proposal that could be implemented by each train at the gate 1+ phase. Having a common ping format still leaves freedom to CD trains to send specific payloads within the pings as every product will have different aspects to be measured.
Next Steps
- Share the basic ping format with all the different CD trains and get their feedback so we can iterate over it and adapt to the potential needs of any train.
- Understand where are the trains in terms of measurement and which kind of metrics they consider important.
- Test the proposal with one train to be used as benchmark
- Help all the teams with Telemetry API adoption, Data Analysis and Reporting.
Program Status
| Milestone | Date | Status |
|---|---|---|
| Connected Devices Ping format drafted | 2/19/2016 | DONE |
| Feedback and comments from Desktop team about CD Ping format addressed | 3/04/2016 | DONE |
| Presentation for Connected Devices Product Management | end of March | ON TARGET |
| London All Hands Conference | 6/2016 |
Development process
Meetings
- Weekly meetings happen on Wednesday at 5 PM UTC (9 AM, PST) in Maria Oteo's Vidyo, notes in Google doc
References
- Telemetry Pipeline
- Common Ping format
- Connected Devices Ping Format proposal
- Connected Devices Metrics Presentation (working on it)
Team
- Dominik Strohmeier - Product Strategy & Insights
- Dylan Oliver - Engineering Manager
- Tamara Hills - Engineering
- Russ Nicoletti - Engineering
- Michael Bryant - QA/Engineering
- Maria Oteo - EPM