Connected Devices/Projects/Metrics: Difference between revisions

Latest revision as of 20:35, 9 December 2016

With ideas becoming more mature during the innovation process - and eventually becoming products - monitoring progress, measuring market fit or understanding the customer lifecycle becomes more and more important for Product teams. This is also known as Innovation Accounting. Even early in the innovation and product development process, testing hypotheses through experiments and creating insights from the data collected is an essential part of the Lean Startup approach.

For all of the Connected Devices products, collecting metrics is essential to guide decision making during the gating process. One problem with the set of Connected Devices products is that these products are going to be very different in terms of functionalities and technologies. Unlike FirefoxOS where one set of metrics was good for all supported devices, we will need very specific solutions for each product that will ideally follow a common structure of Innovation Accounting.

In addition, accessing data from different products should require as low effort as possible for decision makers in Connected Devices (engineers, PMs, senior management…).

Rationale

Nearly all the Connected Devices trains are likely, at some point of time, to require collection of data. Data gathered from the projects is extremely useful to understand how people use our products but also to check how changes implemented in the Products affect how people use them.

In summary The idea of the CD Metrics Framework is providing all the CD trains an easy way to:

Collect metrics in the products
Get Access to the metrics information
Visualize the metrics information

The Framework

After firstly assessing Mozilla Telemetry Pipeline, and later several third party solutions the team decided to go for Google Analytics (GA) as the Framework for Collecting Metrics. The key reasons are:

Technical Fit: Google Analytics meets our needs both in terms of metric collections, accessing to raw data and visualizing it.
Adoption: The integration of GA with CD trains seemed to be the faster-one (e.g. in one day we managed to get some results in one of the trains: SmartHome).
Legally/Commercially: GA has been already vetted in Mozilla and we are already paying for a premium service

It’s important to clarify that the framework is using the Google Analytics Measurement Protocol that is more appropriate for IoT projects than Classic Google Analytics which is more e-Commerce related. The metrics are stored by GA in a Google BigQuery instance.

The Data can be later on retrieved via the Google Analytics Console that is very convenient to check if the data is being received and getting simple charts about the trains (e.g. how many users are currently “live”?).

In addition to the Google Analytics Console, the trains could also use Google Analytics Data Studio.

Furthermore, Re:Dash can be used to formulate SQL queries against the BigQuery Database. Re:Dash is a third party tool that can be used to get more sophisticated reports. Additional tools exist to create reports, that could be used by the CD trains, if they deemed they are necessary.

Additional Tools and Activities

Apart from identifying what is the best toolchain and framework for being used in CD trains, the CD Metrics Team has been working in different activities to ease adoption, ensure consistency and help with the visualisation activities.

Ease adoption: By Implementing a library for Rust and JS that simplifies the adoption of Google Analytics by the trains. (Note: The Rust library could be also used in other programming languages)
Ensure consistency: By documenting the parameters that can be used for collection of events
Ease visualisation: By evaluating Reporting/visualization tools to ensure that metric data can be easily analyzed by the trains and providing some example charts.

In other words, we have done different tasks to ensure all the trains can onboard easily and send information in a consistent way that can be used to get easily dashboards/reports that are also consistent cross-train.

Project Status and onboarding on the metrics framework

CD Metrics is a framework to ease data collection and some tools to ease data manipulation and visualization. The target of the metrics team is not creating reports on behalf of the trains but giving them access to the data and support to getting started with the framework tools. I.e. data ownership relies on the trains themselves: they are the ones that have full access to the data so they can view/query/report as they see fit.

A first version of the framework is already available, so any CD train could start the on boarding now. For doing so, we have developed a set of guidelines to help with the on boarding process to ingrate CD Metrics in your train: CD Metrics Integration

Why Not Telemetry?

The initial approach we explored was leveraging the unified Telemetry Pipeline to send data to the metrics infrastructure and process it. The idea was reusing an architecture already existing and widely used at Mozilla. In that way could rely on the capability the platform has for validating, sanitizing and even automatically analyzing device data.

After some months working to ensure the information sent was consistent, relying on the Telemetry "Ping" mechanism already used in many Mozilla products and defining one common Connected Devices Ping Format proposal that could be implemented by each train, we changed our approach.

The benefits of using Telemetry are clear for Gecko-based projects, but this might not apply to the CD trains:

The CD projects are intended to be initially small projects and Google Analytics and other 3rd party providers offer frameworks with lower learning and adoption curves. Using Telemetry could be an overkill to them.
Gecko and Telemetry are very coupled and some of the Telemetry advantages require Gecko. Indeed, some other projects in Mozilla that are not using Gecko have already started to use alternative solutions for Metrics.

Project Management

We use Trello board for project management. All ongoing tasks are listed there.

Sprint Backlog

Development process

Repositories

Metrics controller: https://github.com/tamarahills/metrics_controller
SQL for views & reports: https://github.com/dylano/metrics

IRC

You can find us on irc.mozilla.org in the #cd-metrics channel.

Methodology

The project is in a very mature state, as the Framework is already been defined and tested with some projects (as project Haiku and project Magnet) so we have reduced the frequency of our meetings/standups... but we maintain our weekly meetings in order check if new requirements emerge or any other project need our help to integrate metrics.

Weekly meeting in CD_Metrics Vidyo room on Monday at 5:30 PM UTC (9:30 AM, PST), notes in Google doc

Calendar

The CD Metrics team has a public calendar with every team meeting Calendar

Adding Metrics to your Project/Train

The following link has instructions and details for adding metrics to your Project/train

Team

Dylan Oliver - Engineering Manager
Tamara Hills - Engineering
Russ Nicoletti - Engineering
Michael Bryant - QA/Engineering
Maria Oteo - EPM

@@ Line 5: / Line 5: @@
 In addition, accessing data from different products should require as low effort as possible for decision makers in Connected Devices (engineers, PMs, senior management…).
-== Project Overview ==
+== Rationale ==
-Our goal is to build a framework that will allow each train to collect the specific data needed, but still use the [http://gecko.readthedocs.org/en/latest/toolkit/components/telemetry/telemetry/index.html unified Telemetry Pipeline] to send data to the metrics infrastructure. So we will be able to rely on architecture that already exists in Mozilla and which has running processes for validating, sanitizing and even automatically analyzing device data.
+Nearly all the Connected Devices trains are likely, at some point of time, to require collection of data. Data gathered from the projects is extremely useful to understand how people use our products but also to check how changes implemented in the Products affect how people use them.
-Having the Telemetry API as a common bearer is not enough as we need to ensure the information sent is consistent. For doing so, we will rely on the [https://gecko.readthedocs.org/en/latest/toolkit/components/telemetry/telemetry/common-ping.html Telemetry "Ping" mechanism] that is already used in many Mozilla products.
+In summary The idea of the CD Metrics Framework is providing all the CD trains an easy way to:
+* Collect metrics in the products
+* Get Access to the metrics information
+* Visualize the metrics information
-In order to ensure information is consistent, can be easily processed and aggregated, we have defined one common [https://docs.google.com/a/mozilla.com/document/d/1nMdnuYWGSXRzLramzzAWpR78ZRA5645bpWAWKIyTvp0/edit?usp=sharing|Connected Connected Devices Ping Format proposal] that could be implemented by each train at the gate 1+ phase.
+== The Framework ==
+After firstly assessing Mozilla Telemetry Pipeline, and later [https://docs.google.com/spreadsheets/d/1zMgrunttlH0qSr-2qpiIAPb6DPo837V68AGYmJYCPME/edit#gid=0 several third party solutions] the team decided to go for Google Analytics (GA) as the Framework for Collecting Metrics. The key reasons are:
+* Technical Fit: Google Analytics meets our needs both in terms of metric collections, accessing to raw data and visualizing it.
+* Adoption: The integration of GA with CD trains seemed to be the faster-one (e.g. in one day we managed to get some results in one of the trains: SmartHome).
+* Legally/Commercially: GA has been already vetted in Mozilla and we are already paying for a premium service
+It’s important to clarify that the framework is using the [https://developers.google.com/analytics/devguides/collection/protocol/v1/ Google Analytics Measurement Protocol] that is more appropriate for IoT projects than Classic Google Analytics which is more e-Commerce related. The metrics are stored by GA in a Google BigQuery instance.
-Connected Devices will be aiming to use a common ping format across all of Connected Devices.  In summary, this currently consists of three ping formats:
+The Data can be later on retrieved via the [https://www.google.com/analytics/360-suite/data-studio/ Google Analytics Console] that is very convenient to check if the data is being received and getting simple charts about the trains (e.g. how many users are currently “live”?).
-* FTU Ping - Typically sent after the device is started for the first time.  This helps to determine how many Active devices there are.
-* Core Ping - Contains histograms that developers have instrumented in their code using the Metrics Library.  Sent in a recurring interval such as once per day or once every two weeks depending on requirements.  Whether or not this is sent is dependent on privacy choice of the user.
-* Crash Ping - Sent when there is a crash.  Also dependent on privacy choice of user.
-These pings will all be submitted to the Telemetry data pipeline at incoming.telemetry.mozilla.org.Having a common ping format still leaves freedom to CD trains to send specific payloads within the pings as every product will have different aspects to be measured.
+In addition to the Google Analytics Console, the trains could also use Google Analytics Data Studio.
-=== Next Steps ===
+Furthermore, [https://sql.telemetry.mozilla.org/ Re:Dash] can be used to formulate SQL queries against the BigQuery Database. Re:Dash is a third party tool that can be used to get more sophisticated reports. Additional tools exist to create reports, that could be used by the CD trains, if they deemed they are necessary.
-* Share the basic ping format with all the different CD trains and get their feedback so we can iterate over it and adapt to the potential needs of any train.
-* Build the common CD framework (Controller and Histograms libraries) that can be used by any of the trains independently of the technology chosen for each of them
-* Understand where are the trains in terms of measurement and which kind of metrics they consider important.
-* Test the proposal with one train to be used as benchmark
-* Help all the teams with Telemetry API adoption, Data Analysis and Reporting.
-== Program Status ==
+== Additional Tools and Activities ==
-{| class="wikitable"
+Apart from identifying what is the best toolchain and framework for being used in CD trains, the CD Metrics Team has been working in different activities to ease adoption, ensure consistency and help with the visualisation activities.
-! style="text-align: center;" | Milestone
-! style="text-align: center;" | Date
-! style="text-align: center;" | Status
-! style="text-align: center;" | Comments
-|-
-| Connected Devices Metrics Library Architecture Design
-| 4/08/2016
-! style="background:#00EC00;" | ON TARGET
-|
-|-
-| Presentation for Connected Devices Product Management
-| End of April
-! style="background:#00EC00;" | ON TARGET
-| Product Management meetup has been delayed, no final date yet but the earlier date would end of April.
-|-
-| CD Metrics Milestones
-|
-! style="background:#C1C6CA;" | NOT STARTED
-| Metrics milestones will be added as milestones once they're identified
-|-
-|
-|
-!
-|
-|-
-| London All Hands Conference Presentation
-| 6/13/2016
-! style="background:#C1C6CA;" | NOT STARTED
-|
-|}
-'''Status Key'''
+* Ease adoption: By Implementing a library for Rust and JS that simplifies the adoption of Google Analytics by the trains. (Note: The Rust library could be also used in other programming languages)
-{| class="wikitable"
+* Ensure consistency: By documenting the parameters that can be used for collection of events
-! style="text-align: center;" | Color
+* Ease visualisation: By evaluating Reporting/visualization tools to ensure that metric data can be easily analyzed by the trains and providing some example charts.
-! style="text-align: center;" | Status
-! style="text-align: center;" | Key
-|-
-! style="background:#00EC00;" |
-| On Target
-| The project or deliverable is expected to meet its due date.
-|-
-! style="background:#FFFF00;" |
-| Challenged
-| The project or deliverable is facing an issue that might cause it to miss its due date, but a “get well” plan has been developed to get it back on track.
-|-
-! style="background:#FF2800;" |
-| At Risk or Late
-| The project or deliverable is blocked or facing an issue that might cause it to miss its due date, and there’s no “get well” plan to get it back on track, or it is already late.
-|-
-! style="background:#00B2FF;" |
-| Done
-| The project or deliverable has been completed.
-|-
-! style="background:#C1C6CA;" |
-| On Hold or Not Started
-| The project or deliverable has either not been started or has been placed on hold.
-|}
-== Archived Completed Deliverables ==
+In other words, we have done different tasks to ensure all the trains can onboard easily and send information in a consistent way that can be used to get easily dashboards/reports that are also consistent cross-train.
-{| class="wikitable"
-! style="text-align: center;" | Milestone
-! style="text-align: center;" | Date
-! style="text-align: center;" | Status
-! style="text-align: center;" | Status Notes
-|-
-| Connected Devices Ping Format
-| 3/04/2016
-! style="background:#00B2FF;" | Done
-| [https://gecko.readthedocs.org/en/latest/toolkit/components/telemetry/telemetry/common-ping.html CD Common Ping format]
-|-
-| Connected Devices Crash Ping Format drafted
-| 3/11/2016
-! style="background:#00B2FF;" | Done
-| [https://docs.google.com/a/mozilla.com/document/d/1nMdnuYWGSXRzLramzzAWpR78ZRA5645bpWAWKIyTvp0/edit?usp=sharing Connected Connected Devices Ping Format proposal]
-|-
-| High level Histograms Architecture Design
-| 4/08/2016
-! style="background:#00B2FF;" | Done
-| [https://docs.google.com/presentation/d/1QXu6j2SknZKqMz2_Xl-Z1b7MiddvNsedth-QWWljJMs/edit#slide=id.p Histograms Architecture Design Diagram]
-|-
-| Connected Devices Dashboard and Analysis Requirements
-| 4/08/2016
-! style="background:#00EC00;" | On target
-| [https://docs.google.com/document/d/1TjKtcnyEiyJ6ix8FRTICKFjJO2O9_AQM4xf8jw7yYdg/edit Connected Devices Dashboard and Analysis Requirements]
-|-
-| Connected Devices Metrics Presentation for Product Management team
-| No final date yet
-! style="background:#00EC00;" | On target
-| [https://docs.google.com/presentation/d/1qAWHPSphYbdqqpBTpx2RK0vf2QtqRAk0hybRsGERYfo/edit#slide=id.p Connected Devices Metrics Presentation] (working on it)
-|}
-== Program Timeline ==
+== Project Status and onboarding on the metrics framework ==
-This timeline is simply a placeholder. Timeline to be inserted once milestones and schedule are known.
+CD Metrics is a framework to ease data collection and some tools to ease data manipulation and visualization. The target of the metrics team is not creating reports on behalf of the trains but giving them access to the data and support to getting started with the framework tools.  I.e. data ownership relies on the trains themselves: they are the ones that have full access to the data so they can view/query/report as they see fit.
-[[File:Release Timeline.png]]
+A first version of the framework is already available, so any CD train could start the on boarding now. For doing so, we have developed a set of guidelines to help with the on boarding process to ingrate CD Metrics in your train: '''[[Connected_Devices/Projects/Metrics/Integration|CD Metrics Integration]]'''
+== Why Not Telemetry? ==
+The initial approach we explored was leveraging the [http://gecko.readthedocs.io/en/latest/toolkit/components/telemetry/telemetry/index.html unified Telemetry Pipeline] to send data to the metrics infrastructure and process it. The idea was reusing an architecture already existing and widely used at Mozilla. In that way could rely on the capability the platform has for validating, sanitizing and even automatically analyzing device data.
+After some months working to ensure the information sent was consistent, relying on the [https://gecko.readthedocs.io/en/latest/toolkit/components/telemetry/telemetry/common-ping.html Telemetry "Ping"] mechanism already used in many Mozilla products and defining one [https://docs.google.com/document/d/1nMdnuYWGSXRzLramzzAWpR78ZRA5645bpWAWKIyTvp0/edit common Connected Devices Ping Format proposal] that could be implemented by each train, we changed our approach.
+The benefits of using Telemetry are clear for Gecko-based projects, but this might not apply to the CD trains:
+* The CD projects are intended to be initially small projects and Google Analytics and other 3rd party providers offer frameworks with lower learning and adoption curves. Using Telemetry could be an overkill to them.
+* Gecko and Telemetry are very coupled and some of the Telemetry advantages require Gecko. Indeed, some other projects in Mozilla that are not using Gecko have already started to use alternative solutions for Metrics.
 == Project Management ==
 We use [https://trello.com/b/rxfeJFkg/cd-metrics-telemetry Trello board] for project management. All ongoing tasks are listed there.
 === Sprint Backlog ===
-*[[Connected Devices/Projects/Metrics/Sprint 2| Sprint 2 (March 28th - April 8th)]] '''Current Sprint'''
+*[[Connected Devices/Projects/Metrics/Sprint 4| Sprint 4 (May 2nd - May 13th)]]
+*[[Connected Devices/Projects/Metrics/Sprint 3| Sprint 3 (April 11st - April 21st)]]
+*[[Connected Devices/Projects/Metrics/Sprint 2| Sprint 2 (March 28th - April 8th)]]
 *[[Connected Devices/Projects/Metrics/Sprint 1| Sprint 1 (March 15th - March 25th)]]
 == Development process ==
 === Repositories ===
-* Yoric's telemetry library: https://github.com/Yoric/telemetry.rs
 * Metrics controller: https://github.com/tamarahills/metrics_controller
+* SQL for views & reports: https://github.com/dylano/metrics
 === IRC ===
@@ Line 139: / Line 67: @@
 === Methodology ===
-The project is in early state but we have planned to use some agile practices:
+The project is in a very mature state, as the Framework is already been defined and tested with some projects (as project Haiku and project Magnet) so we have reduced the frequency of our meetings/standups... but we maintain our weekly meetings in order check if new requirements emerge or any other project need our help to integrate metrics.
-* Weekly meetings in CD_Metrics Vidyo room on Wednesday at 5:15 PM UTC (9:15 AM, PST), notes in [https://docs.google.com/a/mozilla.com/document/d/19vn1f-XXJCwKlNuo1dm2ruRicee6mp3k-WHd2x5AmDM/edit?usp=sharing Google doc]
+* Weekly meeting in CD_Metrics Vidyo room on Monday at 5:30 PM UTC (9:30 AM, PST), notes in [https://docs.google.com/a/mozilla.com/document/d/19vn1f-XXJCwKlNuo1dm2ruRicee6mp3k-WHd2x5AmDM/edit?usp=sharing Google doc]
-* Sprint Planning meeting every other Monday in CD_Metrics Vidyo room at 5:15 PM UTC (9:15 AM, PST)
-* Daily standup meetings everyday except the days with Sprint Planning or Weekly meeting in CD_Metrics Vidyo room at 5:15 PM UTC (9:15 AM, PST). You can include your daily report in this [https://public.etherpad-mozilla.org/p/CD_Metrics_team etherpad] in case you cannot attend the meeting
 ==== Calendar ====
 The CD Metrics team has a public calendar with every team meeting
 [https://www.google.com/calendar/embed?src=mozilla.com_o1oni7fpquevo135uorqld6r68@group.calendar.google.com Calendar]
-===== Instructions for Adding to your Calendar =====
+== Adding Metrics to your Project/Train ==
-# Open the [https://www.google.com/calendar/embed?src=mozilla.com_o1oni7fpquevo135uorqld6r68@group.calendar.google.com calendar].
+The following link has instructions and details for adding metrics to your Project/train
-# Click on the "+ Google Calendar" button in the very bottom right of your screen.
+*[[Connected Devices/Projects/Metrics/Integration| Integration]]
+*[[Connected Devices/Projects/Metrics/Database_and_Visualization | Database And Visualization ]]
-You can also use the [https://www.google.com/calendar/feeds/mozilla.com_g1v696sr9vu27f2tuvunmevna0%40group.calendar.google.com/public/basic XML] and [https://www.google.com/calendar/ical/mozilla.com_g1v696sr9vu27f2tuvunmevna0%40group.calendar.google.com/public/basic.ics ICS] methods, but these are not recommended.
-Note: The "Find a Time" feature will not work for other people if you import this calendar. As a consequence, others will not see that you are unavailable when attending a FlyWeb meeting. We suggest either accepting this, or adding the meetings to your main calendar as well.
-== References ==
-* [http://gecko.readthedocs.org/en/latest/toolkit/components/telemetry/telemetry/index.html Telemetry Pipeline]
-* [https://gecko.readthedocs.org/en/latest/toolkit/components/telemetry/telemetry/common-ping.html Common Ping format]
 == Team ==