Connected Devices/Projects/Metrics

With ideas becoming more mature during the innovation process - and eventually becoming products - monitoring progress, measuring market fit or understanding the customer lifecycle becomes more and more important for Product teams. This is also known as Innovation Accounting. Even early in the innovation and product development process, testing hypotheses through experiments and creating insights from the data collected is an essential part of the Lean Startup approach.

For all of the Connected Devices products, collecting metrics is essential to guide decision making during the gating process. One problem with the set of Connected Devices products is that these products are going to be very different in terms of functionalities and technologies. Unlike FirefoxOS where one set of metrics was good for all supported devices, we will need very specific solutions for each product that will ideally follow a common structure of Innovation Accounting.

In addition, accessing data from different products should require as low effort as possible for decision makers in Connected Devices (engineers, PMs, senior management…).

Project Overview

Our goal is to build a framework that will allow each train to collect the specific data needed, but still use the unified Telemetry Pipeline to send data to the metrics infrastructure. So we will be able to rely on architecture that already exists in Mozilla and which has running processes for validating, sanitizing and even automatically analyzing device data.

Having the Telemetry API as a common bearer is not enough as we need to ensure the information sent is consistent. For doing so, we will rely on the Telemetry "Ping" mechanism that is already used in many Mozilla products.

In order to ensure information is consistent, can be easily processed and aggregated, we have defined one common Connected Devices Ping Format proposal that could be implemented by each train at the gate 1+ phase. Having a common ping format still leaves freedom to CD trains to send specific payloads within the pings as every product will have different aspects to be measured.

Next Steps

• Share the basic ping format with all the different CD trains and get their feedback so we can iterate over it and adapt to the potential needs of any train.
• Understand where are the trains in terms of measurement and which kind of metrics they consider important.
• Test the proposal with one train to be used as benchmark
• Help all the teams with Telemetry API adoption, Data Analysis and Reporting.

Program Status

Milestone	Date	Status
Connected Devices Ping format drafted	2/19/2016	DONE
Feedback and comments from Desktop team about CD Ping format addressed	3/04/2016	DONE
Presentation for Connected Devices Product Management	end of March	ON TARGET
London All Hands Conference	6/2016

Status Key

Color	Status	Key
	On Target	The project or deliverable is expected to meet its due date.
	Challenged	The project or deliverable is facing an issue that might cause it to miss its due date, but a “get well” plan has been developed to get it back on track.
	At Risk or Late	The project or deliverable is blocked or facing an issue that might cause it to miss its due date, and there’s no “get well” plan to get it back on track, or it is already late.
	Done	The project or deliverable has been completed.
	On Hold or Not Started	The project or deliverable has either not been started or has been placed on hold.

---

Development process

• Trello board

Meetings

• Weekly meetings happen on Wednesday at 5 PM UTC (9 AM, PST) in Maria Oteo's Vidyo, notes in Google doc

References

• Telemetry Pipeline
• Common Ping format
• Connected Devices Ping Format proposal
• Connected Devices Metrics Presentation (working on it)

Team

• Dominik Strohmeier - Product Strategy & Insights
• Dylan Oliver - Engineering Manager
• Tamara Hills - Engineering
• Russ Nicoletti - Engineering
• Michael Bryant - QA/Engineering
• Maria Oteo - EPM