Firefox OS/Metrics: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Update the team)
 
(205 intermediate revisions by 8 users not shown)
Line 1: Line 1:
This is a placeholder page for design / discussion around Metrics in FxOS
=Goals=
=Phase 1=
The FxOS Metrics project enables making data driven decisions for the following: 
==Requirements==
*Understanding how FxOS is doing in the market versus product/business goals
#For User Benefit, knowing the following information will security updates and app functionality
*Gaining insights from quantitative and qualitative data, to enable improve the product/business goals
## Build IDs, Country, Operator, OEM, Version of HW: These are needed for Device Updates and Security patches: When a bug is found, or a security risk is detected, they need to be sent as an update to the particular OEM, Operator.
*Understanding usage models, to improve future product development and opportunities
##Screen Height, Width, Pixel density: This is needed for CoreApps development (Gallery, Camera, Dialer, Browser) and 3rd party apps. These apps need to know the screen sizes, resolutions that they would need to support.
*Determining effectiveness of marketing campaigns.
#Product Planning: Need number of devices sold. This will help product decisions be data driven.


==Design of Activation Ping==
=Metrics=
#Is triggered through the first-time use experience / activation. Will automatically retry until it is successful (on a 1 hour interval)
{| class="wikitable"
#Never pings again once it is successful (unless the phone's "userdata" partition is formatted through the recovery partition, but this is not a case we support generally, it means wiping all of the user's data)
|-
#Can also be disabled, but requires extra work on top of disabling app updates or removing the Firefox marketplace app
! Customer lifecycle!! Metrics to be collected !! Dashboard
===Delta to what exists today===
|-
#Information:
|Acquiring Customers
##ActivationPing Gives the total number of "sold"(activated) FFOS devices, whereas current ADI ping gives the devices connected to WiFi on a particular day
|
###Product decision making needs the information about devices sold (ActivatioPing). Current ADI ping approximated devices active on a certain day on WiFi. Difference is retention. This will help determine focus on what is causing the dropoff, investigate certain markets are doing better, and making investments in improving the retention
*Partners/Channels though which devices are sold
###Activation Ping has additional information on Build IDs, Country, Operator, OEM, Version of HW, Screen Size, Screen Height, Width, Pixel density
|[https://metrics.mozilla.com/protected/dashboards/fxos-ftu/ Dashboard]: Choose Operator/OEM/Country combination of channel
####ADI ping has model of device info.
|-
#Reliability
|
##Current Ping from 1.3 shows approx 5 devices reporting as of March 24th 2014. https://dataviz.mozilla.org/views/ADI_Ping_1_3/HistoricalADITrend#2
Activation of Devices
##From previous versions, Partners removed ADI ping due to cost concern. Activation Ping is sent once in a lifetime, which should allay cost concerns.
|
*Total number of Activated devices
|[https://metrics.mozilla.com/protected/dashboards/fxos-ftu/ Dashboard]: Choose 'All' in SW/Operator/OEM/ to get a count
|-
|
Retention Of Customers
|
*MAU (Monthly Active Users)
*Apps usage per month
*Performance (App Startup time, Power consumption)
|Internal
|-
|Revenue
|
*Number of searches done
*Paid App Downloads
|Internal
|-
|Referral
|
*Net Promoter Score
**Recommendations from Users
**User Satisfaction Factor
|Future
|}


== ==
=Roadmap=
{| class="wikitable"
|-
!FxOS Release !! Metrics to be collected !! Bugs
|-
|1.3
|
*Acquisition, Activations:
**[https://wiki.mozilla.org/Firefox_OS/Metrics/activationping Activation Ping]:
|
*Engg: 986701,997960
*Legal: 980920
*Privacy: 992487
|-
|2.1
|
*Retention
**[https://wiki.mozilla.org/FirefoxOS/Metrics/App_Usage App Usage (Default Off)]
|
*Client: 982663
*Legal: 1021259, 1109429
|-
|2.2
|
*Retention
**[https://wiki.mozilla.org/FirefoxOS/Metrics/App_Usage App Usage (Default On)]
***[https://wiki.mozilla.org/FirefoxOS/Metrics/App_Usage AppUsage],
***[https://docs.google.com/a/mozilla.com/document/d/1i9CwaeLsOM-XS1Z8UZ88-b03CkPyxHI07sLhcZ9z4EY/edit#heading=h.lqni56rfu030 MAU]
*Acquisition:
**[https://bugzilla.mozilla.org/1021173 Report Preinstalled Apps]
*Revenue
**[https://bugzilla.mozilla.org/show_bug.cgi?id=1126524 Num of Searches/Revenue ]
|
*[https://wiki.mozilla.org/FirefoxOS/Metrics/App_Usage AppUsage], MAU:
**[https://bugzilla.mozilla.org/1109422 1109422 Client side work for enabling AppUsage ],
**[https://bugzilla.mozilla.org/1119009 1119009 Marketplace Apps Only]
**[https://bugzilla.mozilla.org/1109426 1109426 Server side dashboard for Most used apps: ]
*[https://bugzilla.mozilla.org/1021173 1021173 Report Preinstalled Apps, ]
*[https://bugzilla.mozilla.org/1126524 1126524 Search ]
|-
|2.5
|
*Retention:
**[https://bugzilla.mozilla.org/1179004 OTA collection of App Startup time (1179004)].
*Activation
**Accuracy of activation: [https://bugzilla.mozilla.org/1171685 FTU Ping: Avoid counting repeat pings from a known device. (1171685).]
*Privacy:
**[https://bugzilla.mozilla.org/1181295 Converged UX for opt-in, opt-out of metrics. 1181295.]
*Scaling:
**[https://bugzilla.mozilla.org/1152000 Framework (Metabug: 1152000) to collect custom Metrics and histogram]
**Non Phone devices support: [https://bugzilla.mozilla.org/1171685 Using TV serial number to avoid counting repeat pings. 1171685.]
**B2GDroid Support ([https://bugzilla.mozilla.org/show_bug.cgi?id=1198338 B2GDroid Metric Collection 1198338])
**AddOns Support ([https://bugzilla.mozilla.org/1198346: AddOns Metric collection 1198346])
|
|-
|2.6 (May 2016)
| (In Planning [https://bugzilla.mozilla.org/show_bug.cgi?id=1226182 Meta Bug for 2.6 (1226182)])
*Activation
**Optimizing FTU: % of signups to Moz services: FxAccounts, FMD
**Accuracy of Retention/MAU: [https://bugzilla.mozilla.org/1198164 FTU ping should include info on whether User opt-in for metrics (1198164)]
*Retention
**[https://bugzilla.mozilla.org/show_bug.cgi?id=1226458 Support for progressive web apps (1226458)]
**[https://bugzilla.mozilla.org/show_bug.cgi?id=1226274 Make uuid available to Foxfooder app (1226274)]
**Inclusion of device metrics in A/B testing (likely to be moved beyond 2.5)
**Engagement with device: [https://bugzilla.mozilla.org/1193225 Number of voice calls, data (wifi and cellular), number of text messages. 1193225.]
** Device Startup time: [https://bugzilla.mozilla.org/1177226 OTA collection of device dtartup time (1177226)]
**Advanced A/B testing
**Churn Rate & User Lifecycle Analysis (In planning)
**Framework Improvements
***Enum Type and Boolean Type histograms
**Support for New Security model [https://bugzilla.mozilla.org/show_bug.cgi?id=1225960 1225960]
**FoTA Ping [https://bugzilla.mozilla.org/show_bug.cgi?id=1202970 Engg bug: 1202970], [https://bugzilla.mozilla.org/show_bug.cgi?id=1202987. Legal bug: 1202987].
*Referral
**Net Promoter Score (User Satisfaction Factor)
*Privacy
**[https://bugzilla.mozilla.org/1126513 User ability to delete collected info. 1126513.]
*Server
**[https://bugzilla.mozilla.org/show_bug.cgi?id=1226062 Graphing of Addon Histograms]
**[https://bugzilla.mozilla.org/show_bug.cgi?id=1226064 Graphing of Keyed Histograms]
|-
|End-2016 (In Planning)
|
*Retention
**Content Recommendation Engine based on usage (User Benefit)
|
|}


=Phase 2=
=User Stories=
==Requirements==
*[https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AmVPziKCVEUEdGFoczVCaHBzOGxQcXNxWWs3aWRSalE#gid=0 User Stories]
 
= FoxFooding =
 
[https://wiki.mozilla.org/FirefoxOS/Metrics/FoxFooding Data gathered for the FoxFooding program]
 
=Virtual Team=
*Device Engg:  Tamara Hills, Gregor Wagner, Russ Nicoletti
*Device QA: Shing Lyu, Iris Hsiao, Gerry Chang
*Server Metrics: Dave Zeber, John Jensen
*Planning: Dominik Strohmeier, Maria Oteo
*Product Mgmt: Peter Dolanjski, Maria Oteo
*Alumni: Marshall Culpepper, Hema Koka
 
=Links=
*[https://wiki.mozilla.org/FirefoxOS/Metrics/CustomMetrics CustomMetrics]
*[https://wiki.mozilla.org/FirefoxOS/Metrics/EngineeringMetrics EngineeringMetrics]
*[https://wiki.mozilla.org/FirefoxOS/Metrics/activationping Activation Ping]
*[https://wiki.mozilla.org/FirefoxOS/Metrics/App_Usage AppUsage Data]
*[https://docs.google.com/a/mozilla.com/document/d/1i9CwaeLsOM-XS1Z8UZ88-b03CkPyxHI07sLhcZ9z4EY/edit#heading=h.lqni56rfu030 MAU Doc]
*[https://wiki.mozilla.org/FirefoxOS/Metrics Main Metrics Page (this page)]
*[https://wiki.mozilla.org/B2G/QA/Ping_Testing Testing Guide]
*[https://wiki.mozilla.org/Firefox_OS/Metrics/FAQ Frequently Asked Questions]
*[https://docs.google.com/document/d/1CxpSQzjvV-N3owFHpOWcki9dmNZa874-x2MJp4NxG-g/edit#heading=h.jgf7qkensd70 Performance Goals]

Latest revision as of 16:42, 21 January 2016

Goals

The FxOS Metrics project enables making data driven decisions for the following:

  • Understanding how FxOS is doing in the market versus product/business goals
  • Gaining insights from quantitative and qualitative data, to enable improve the product/business goals
  • Understanding usage models, to improve future product development and opportunities
  • Determining effectiveness of marketing campaigns.

Metrics

Customer lifecycle Metrics to be collected Dashboard
Acquiring Customers
  • Partners/Channels though which devices are sold
 Dashboard: Choose Operator/OEM/Country combination of channel

Activation of Devices

  • Total number of Activated devices
 Dashboard: Choose 'All' in SW/Operator/OEM/ to get a count

Retention Of Customers

  • MAU (Monthly Active Users)
  • Apps usage per month
  • Performance (App Startup time, Power consumption)
 Internal
Revenue
  • Number of searches done
  • Paid App Downloads
 Internal
Referral
  • Net Promoter Score
    • Recommendations from Users
    • User Satisfaction Factor
 Future

Roadmap

FxOS Release Metrics to be collected Bugs
1.3
  • Engg: 986701,997960
  • Legal: 980920
  • Privacy: 992487
2.1
  • Client: 982663
  • Legal: 1021259, 1109429
2.2
2.5
2.6 (May 2016) (In Planning Meta Bug for 2.6 (1226182))
End-2016 (In Planning)
  • Retention
    • Content Recommendation Engine based on usage (User Benefit)

User Stories

FoxFooding

Data gathered for the FoxFooding program

Virtual Team

  • Device Engg: Tamara Hills, Gregor Wagner, Russ Nicoletti
  • Device QA: Shing Lyu, Iris Hsiao, Gerry Chang
  • Server Metrics: Dave Zeber, John Jensen
  • Planning: Dominik Strohmeier, Maria Oteo
  • Product Mgmt: Peter Dolanjski, Maria Oteo
  • Alumni: Marshall Culpepper, Hema Koka

Links

Retrieved from "https://wiki.allizom.org/index.php?title=Firefox_OS/Metrics&oldid=1113701"