Firefox OS/Metrics: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Update the team)
 
(210 intermediate revisions by 8 users not shown)
Line 1: Line 1:
This is a placeholder page for design / discussion around Metrics in FxOS
=Goals=
=Phase 1=
The FxOS Metrics project enables making data driven decisions for the following: 
==Requirements==
*Understanding how FxOS is doing in the market versus product/business goals
For User Benefit, knowing the following information will security updates and app functionality
*Gaining insights from quantitative and qualitative data, to enable improve the product/business goals
# Build IDs, Country, Operator, OEM, Version of HW: These are needed for Device Updates and Security patches: When a bug is found, or a security risk is detected, they need to be sent as an update to the particular OEM, Operator.
*Understanding usage models, to improve future product development and opportunities
#Screen Height, Width, Pixel density: This is needed for CoreApps development (Gallery, Camera, Dialer, Browser) and 3rd party apps. These apps need to know the screen sizes, resolutions that they would need to support.
*Determining effectiveness of marketing campaigns.


==Design of Ping==
=Metrics=
#Is triggered through the first-time use experience / activation. Will automatically retry until it is successful (on a 1 hour interval)
{| class="wikitable"
#Never pings again once it is successful (unless the phone's "userdata" partition is formatted through the recovery partition, but this is not a case we support generally, it means wiping all of the user's data)
|-
#Can also be disabled, but requires extra work on top of disabling app updates or removing the Firefox marketplace app
! Customer lifecycle!! Metrics to be collected !! Dashboard
===Delta to what exists today===
|-
#Information:
|Acquiring Customers
## Gives the total number of "sold"(activated) FFOS devices, whereas current ping gives the devices connected to WiFi on a particular day
|
###Product decision making needs information
*Partners/Channels though which devices are sold
#### Devices sold
|[https://metrics.mozilla.com/protected/dashboards/fxos-ftu/ Dashboard]: Choose Operator/OEM/Country combination of channel
#### Devices active
|-
This will help determine focus on what is causing the dropoff, why certain markets are doing better, and making investments in improving the retention
|
#Reliability
Activation of Devices
##Current Ping from 1.3 shows approx 5 devices reporting as of March 24th 2014. https://dataviz.mozilla.org/views/ADI_Ping_1_3/HistoricalADITrend#2
|
##From previous versions, Partners removed due to cost concern
*Total number of Activated devices
## Current ping is sent everyday, while the proposed ping is sent once in a lifetime, which should allay cost concerns
|[https://metrics.mozilla.com/protected/dashboards/fxos-ftu/ Dashboard]: Choose 'All' in SW/Operator/OEM/ to get a count
|-
|
Retention Of Customers
|
*MAU (Monthly Active Users)
*Apps usage per month
*Performance (App Startup time, Power consumption)
|Internal
|-
|Revenue
|
*Number of searches done
*Paid App Downloads
|Internal
|-
|Referral
|
*Net Promoter Score
**Recommendations from Users
**User Satisfaction Factor
|Future
|}


== ==
=Roadmap=
{| class="wikitable"
|-
!FxOS Release !! Metrics to be collected !! Bugs
|-
|1.3
|
*Acquisition, Activations:
**[https://wiki.mozilla.org/Firefox_OS/Metrics/activationping Activation Ping]:
|
*Engg: 986701,997960
*Legal: 980920
*Privacy: 992487
|-
|2.1
|
*Retention
**[https://wiki.mozilla.org/FirefoxOS/Metrics/App_Usage App Usage (Default Off)]
|
*Client: 982663
*Legal: 1021259, 1109429
|-
|2.2
|
*Retention
**[https://wiki.mozilla.org/FirefoxOS/Metrics/App_Usage App Usage (Default On)]
***[https://wiki.mozilla.org/FirefoxOS/Metrics/App_Usage AppUsage],
***[https://docs.google.com/a/mozilla.com/document/d/1i9CwaeLsOM-XS1Z8UZ88-b03CkPyxHI07sLhcZ9z4EY/edit#heading=h.lqni56rfu030 MAU]
*Acquisition:
**[https://bugzilla.mozilla.org/1021173 Report Preinstalled Apps]
*Revenue
**[https://bugzilla.mozilla.org/show_bug.cgi?id=1126524 Num of Searches/Revenue ]
|
*[https://wiki.mozilla.org/FirefoxOS/Metrics/App_Usage AppUsage], MAU:
**[https://bugzilla.mozilla.org/1109422 1109422 Client side work for enabling AppUsage ],
**[https://bugzilla.mozilla.org/1119009 1119009 Marketplace Apps Only]
**[https://bugzilla.mozilla.org/1109426 1109426 Server side dashboard for Most used apps: ]
*[https://bugzilla.mozilla.org/1021173 1021173 Report Preinstalled Apps, ]
*[https://bugzilla.mozilla.org/1126524 1126524 Search ]
|-
|2.5
|
*Retention:
**[https://bugzilla.mozilla.org/1179004 OTA collection of App Startup time (1179004)].
*Activation
**Accuracy of activation: [https://bugzilla.mozilla.org/1171685 FTU Ping: Avoid counting repeat pings from a known device. (1171685).]
*Privacy:
**[https://bugzilla.mozilla.org/1181295 Converged UX for opt-in, opt-out of metrics. 1181295.]
*Scaling:
**[https://bugzilla.mozilla.org/1152000 Framework (Metabug: 1152000) to collect custom Metrics and histogram]
**Non Phone devices support: [https://bugzilla.mozilla.org/1171685 Using TV serial number to avoid counting repeat pings. 1171685.]
**B2GDroid Support ([https://bugzilla.mozilla.org/show_bug.cgi?id=1198338 B2GDroid Metric Collection 1198338])
**AddOns Support ([https://bugzilla.mozilla.org/1198346: AddOns Metric collection 1198346])
|
|-
|2.6 (May 2016)
| (In Planning [https://bugzilla.mozilla.org/show_bug.cgi?id=1226182 Meta Bug for 2.6 (1226182)])
*Activation
**Optimizing FTU: % of signups to Moz services: FxAccounts, FMD
**Accuracy of Retention/MAU: [https://bugzilla.mozilla.org/1198164 FTU ping should include info on whether User opt-in for metrics (1198164)]
*Retention
**[https://bugzilla.mozilla.org/show_bug.cgi?id=1226458 Support for progressive web apps (1226458)]
**[https://bugzilla.mozilla.org/show_bug.cgi?id=1226274 Make uuid available to Foxfooder app (1226274)]
**Inclusion of device metrics in A/B testing (likely to be moved beyond 2.5)
**Engagement with device: [https://bugzilla.mozilla.org/1193225 Number of voice calls, data (wifi and cellular), number of text messages. 1193225.]
** Device Startup time: [https://bugzilla.mozilla.org/1177226 OTA collection of device dtartup time (1177226)]
**Advanced A/B testing
**Churn Rate & User Lifecycle Analysis (In planning)
**Framework Improvements
***Enum Type and Boolean Type histograms
**Support for New Security model [https://bugzilla.mozilla.org/show_bug.cgi?id=1225960 1225960]
**FoTA Ping [https://bugzilla.mozilla.org/show_bug.cgi?id=1202970 Engg bug: 1202970], [https://bugzilla.mozilla.org/show_bug.cgi?id=1202987. Legal bug: 1202987].
*Referral
**Net Promoter Score (User Satisfaction Factor)
*Privacy
**[https://bugzilla.mozilla.org/1126513 User ability to delete collected info. 1126513.]
*Server
**[https://bugzilla.mozilla.org/show_bug.cgi?id=1226062 Graphing of Addon Histograms]
**[https://bugzilla.mozilla.org/show_bug.cgi?id=1226064 Graphing of Keyed Histograms]
|-
|End-2016 (In Planning)
|
*Retention
**Content Recommendation Engine based on usage (User Benefit)
|
|}


=Phase 2=
=User Stories=
==Requirements==
*[https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AmVPziKCVEUEdGFoczVCaHBzOGxQcXNxWWs3aWRSalE#gid=0 User Stories]
 
= FoxFooding =
 
[https://wiki.mozilla.org/FirefoxOS/Metrics/FoxFooding Data gathered for the FoxFooding program]
 
=Virtual Team=
*Device Engg:  Tamara Hills, Gregor Wagner, Russ Nicoletti
*Device QA: Shing Lyu, Iris Hsiao, Gerry Chang
*Server Metrics: Dave Zeber, John Jensen
*Planning: Dominik Strohmeier, Maria Oteo
*Product Mgmt: Peter Dolanjski, Maria Oteo
*Alumni: Marshall Culpepper, Hema Koka
 
=Links=
*[https://wiki.mozilla.org/FirefoxOS/Metrics/CustomMetrics CustomMetrics]
*[https://wiki.mozilla.org/FirefoxOS/Metrics/EngineeringMetrics EngineeringMetrics]
*[https://wiki.mozilla.org/FirefoxOS/Metrics/activationping Activation Ping]
*[https://wiki.mozilla.org/FirefoxOS/Metrics/App_Usage AppUsage Data]
*[https://docs.google.com/a/mozilla.com/document/d/1i9CwaeLsOM-XS1Z8UZ88-b03CkPyxHI07sLhcZ9z4EY/edit#heading=h.lqni56rfu030 MAU Doc]
*[https://wiki.mozilla.org/FirefoxOS/Metrics Main Metrics Page (this page)]
*[https://wiki.mozilla.org/B2G/QA/Ping_Testing Testing Guide]
*[https://wiki.mozilla.org/Firefox_OS/Metrics/FAQ Frequently Asked Questions]
*[https://docs.google.com/document/d/1CxpSQzjvV-N3owFHpOWcki9dmNZa874-x2MJp4NxG-g/edit#heading=h.jgf7qkensd70 Performance Goals]

Latest revision as of 16:42, 21 January 2016

Goals

The FxOS Metrics project enables making data driven decisions for the following:

  • Understanding how FxOS is doing in the market versus product/business goals
  • Gaining insights from quantitative and qualitative data, to enable improve the product/business goals
  • Understanding usage models, to improve future product development and opportunities
  • Determining effectiveness of marketing campaigns.

Metrics

Customer lifecycle Metrics to be collected Dashboard
Acquiring Customers
  • Partners/Channels though which devices are sold
 Dashboard: Choose Operator/OEM/Country combination of channel

Activation of Devices

  • Total number of Activated devices
 Dashboard: Choose 'All' in SW/Operator/OEM/ to get a count

Retention Of Customers

  • MAU (Monthly Active Users)
  • Apps usage per month
  • Performance (App Startup time, Power consumption)
 Internal
Revenue
  • Number of searches done
  • Paid App Downloads
 Internal
Referral
  • Net Promoter Score
    • Recommendations from Users
    • User Satisfaction Factor
 Future

Roadmap

FxOS Release Metrics to be collected Bugs
1.3
  • Engg: 986701,997960
  • Legal: 980920
  • Privacy: 992487
2.1
  • Client: 982663
  • Legal: 1021259, 1109429
2.2
2.5
2.6 (May 2016) (In Planning Meta Bug for 2.6 (1226182))
End-2016 (In Planning)
  • Retention
    • Content Recommendation Engine based on usage (User Benefit)

User Stories

FoxFooding

Data gathered for the FoxFooding program

Virtual Team

  • Device Engg: Tamara Hills, Gregor Wagner, Russ Nicoletti
  • Device QA: Shing Lyu, Iris Hsiao, Gerry Chang
  • Server Metrics: Dave Zeber, John Jensen
  • Planning: Dominik Strohmeier, Maria Oteo
  • Product Mgmt: Peter Dolanjski, Maria Oteo
  • Alumni: Marshall Culpepper, Hema Koka

Links

Retrieved from "https://wiki.allizom.org/index.php?title=Firefox_OS/Metrics&oldid=1113701"