Firefox OS/Metrics: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
(Update the team)
 
(188 intermediate revisions by 8 users not shown)
Line 1: Line 1:
=FTU ping=
=Goals=
The FxOS Metrics project enables making data driven decisions for the following: 
*Understanding how FxOS is doing in the market versus product/business goals
*Gaining insights from quantitative and qualitative data, to enable improve the product/business goals
*Understanding usage models, to improve future product development and opportunities
*Determining effectiveness of marketing campaigns.


When a device is first activated, The FTU ping service sends a simple one-off HTTP request to our telemetry server. The HTTP request contains a JSON payload with some high level information about the device and network, no unique hardware identifiers.
=Metrics=
{| class="wikitable"
|-
! Customer lifecycle!! Metrics to be collected !! Dashboard
|-
|Acquiring Customers
|
*Partners/Channels though which devices are sold
|[https://metrics.mozilla.com/protected/dashboards/fxos-ftu/ Dashboard]: Choose Operator/OEM/Country combination of channel
|-
|
Activation of Devices
|
*Total number of Activated devices
|[https://metrics.mozilla.com/protected/dashboards/fxos-ftu/ Dashboard]: Choose 'All' in SW/Operator/OEM/ to get a count
|-
|
Retention Of Customers
|
*MAU (Monthly Active Users)
*Apps usage per month
*Performance (App Startup time, Power consumption)
|Internal
|-
|Revenue
|
*Number of searches done
*Paid App Downloads
|Internal
|-
|Referral
|
*Net Promoter Score
**Recommendations from Users
**User Satisfaction Factor
|Future
|}


Currently, the logic for the FTU ping is broken down like this:
=Roadmap=
# Unless the setting <tt>ftu.pingEnabled</tt> is explicitly set to false, start the FTU ping service. If the setting is missing, or any other value, then the ping service is enabled.
{| class="wikitable"
# Lazily get or create a randomly generated pingID, which is used for server side filtering. This pingID is only generated once per device, and is discarded on the server side.
|-
# Record the time the ping service started, to track the delta of time to a successful ping.
!FxOS Release !! Metrics to be collected !! Bugs
# Pull all relevant ping payload settings asynchronously.
|-
# Create an interval timer to attempt a ping once an hour. This is configurable with <tt>ftu.pingTryInterval</tt>.
|1.3
## Check for availability of voice network information. If none is available, try again on the next interval. If this fails 24 times, then continue trying the ping until is successful. This is configurable in the setting <tt>ftu.pingMaxNetworkFails</tt>.
|
## Send an HTTP request to <tt>FTU_PING_URL/PING_ID/ftu/FirefoxOS/VERSION/UPDATE_CHANNEL/BUILD_ID</tt>. By default, the ping URL is https://fxos.telemetry.mozilla.org, and is configurable in the <tt>ftu.pingURL</tt> setting, or via make with <tt>FTU_PING_URL</tt>.
*Acquisition, Activations:
### If the HTTP request succeeds, set <tt>ftu.pingEnabled</tt> to false, and disable the interval timer. Otherwise, try again on the next interval.
**[https://wiki.mozilla.org/Firefox_OS/Metrics/activationping Activation Ping]:
|
*Engg: 986701,997960
*Legal: 980920
*Privacy: 992487
|-
|2.1
|
*Retention
**[https://wiki.mozilla.org/FirefoxOS/Metrics/App_Usage App Usage (Default Off)]
|
*Client: 982663
*Legal: 1021259, 1109429
|-
|2.2
|
*Retention
**[https://wiki.mozilla.org/FirefoxOS/Metrics/App_Usage App Usage (Default On)]
***[https://wiki.mozilla.org/FirefoxOS/Metrics/App_Usage AppUsage],  
***[https://docs.google.com/a/mozilla.com/document/d/1i9CwaeLsOM-XS1Z8UZ88-b03CkPyxHI07sLhcZ9z4EY/edit#heading=h.lqni56rfu030 MAU]
*Acquisition:
**[https://bugzilla.mozilla.org/1021173 Report Preinstalled Apps]
*Revenue
**[https://bugzilla.mozilla.org/show_bug.cgi?id=1126524 Num of Searches/Revenue ]
|
*[https://wiki.mozilla.org/FirefoxOS/Metrics/App_Usage AppUsage], MAU:
**[https://bugzilla.mozilla.org/1109422 1109422 Client side work for enabling AppUsage ],
**[https://bugzilla.mozilla.org/1119009 1119009 Marketplace Apps Only]
**[https://bugzilla.mozilla.org/1109426 1109426 Server side dashboard for Most used apps: ]
*[https://bugzilla.mozilla.org/1021173 1021173 Report Preinstalled Apps, ]
*[https://bugzilla.mozilla.org/1126524 1126524 Search ]
|-
|2.5
|
*Retention:
**[https://bugzilla.mozilla.org/1179004 OTA collection of App Startup time (1179004)].
*Activation
**Accuracy of activation: [https://bugzilla.mozilla.org/1171685 FTU Ping: Avoid counting repeat pings from a known device. (1171685).]
*Privacy:
**[https://bugzilla.mozilla.org/1181295 Converged UX for opt-in, opt-out of metrics. 1181295.]
*Scaling:
**[https://bugzilla.mozilla.org/1152000 Framework (Metabug: 1152000) to collect custom Metrics and histogram]
**Non Phone devices support: [https://bugzilla.mozilla.org/1171685 Using TV serial number to avoid counting repeat pings. 1171685.]
**B2GDroid Support ([https://bugzilla.mozilla.org/show_bug.cgi?id=1198338 B2GDroid Metric Collection 1198338])
**AddOns Support ([https://bugzilla.mozilla.org/1198346: AddOns Metric collection 1198346])
|
|-
|2.6 (May 2016)
| (In Planning [https://bugzilla.mozilla.org/show_bug.cgi?id=1226182 Meta Bug for 2.6 (1226182)])
*Activation
**Optimizing FTU: % of signups to Moz services: FxAccounts, FMD
**Accuracy of Retention/MAU: [https://bugzilla.mozilla.org/1198164 FTU ping should include info on whether User opt-in for metrics (1198164)]
*Retention
**[https://bugzilla.mozilla.org/show_bug.cgi?id=1226458 Support for progressive web apps (1226458)]
**[https://bugzilla.mozilla.org/show_bug.cgi?id=1226274 Make uuid available to Foxfooder app (1226274)]
**Inclusion of device metrics in A/B testing (likely to be moved beyond 2.5)
**Engagement with device: [https://bugzilla.mozilla.org/1193225 Number of voice calls, data (wifi and cellular), number of text messages. 1193225.]
** Device Startup time: [https://bugzilla.mozilla.org/1177226 OTA collection of device dtartup time (1177226)]
**Advanced A/B testing
**Churn Rate & User Lifecycle Analysis (In planning)
**Framework Improvements
***Enum Type and Boolean Type histograms
**Support for New Security model [https://bugzilla.mozilla.org/show_bug.cgi?id=1225960 1225960]
**FoTA Ping [https://bugzilla.mozilla.org/show_bug.cgi?id=1202970 Engg bug: 1202970], [https://bugzilla.mozilla.org/show_bug.cgi?id=1202987. Legal bug: 1202987].
*Referral
**Net Promoter Score (User Satisfaction Factor)
*Privacy
**[https://bugzilla.mozilla.org/1126513 User ability to delete collected info. 1126513.]
*Server
**[https://bugzilla.mozilla.org/show_bug.cgi?id=1226062 Graphing of Addon Histograms]
**[https://bugzilla.mozilla.org/show_bug.cgi?id=1226064 Graphing of Keyed Histograms]
|-
|End-2016 (In Planning)
|
*Retention
**Content Recommendation Engine based on usage (User Benefit)
|
|}


==Example payload and URL==
=User Stories=
*[https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AmVPziKCVEUEdGFoczVCaHBzOGxQcXNxWWs3aWRSalE#gid=0 User Stories]
<tt>POST https://fxos.telemetry.mozilla.org/submit/telemetry/e426da9f-2a29-4e09-895b-c883903956cb/ftu/FirefoxOS/31.0a1/default/20140325104133</tt>


<code><pre>
= FoxFooding =
{
    "activationTime": 1395769944966,
    "devicePixelRatio": 1,
    "deviceinfo.firmware_revision": "",
    "deviceinfo.hardware": "qcom",
    "deviceinfo.os": "1.5.0.0-prerelease",
    "deviceinfo.platform_build_id": "20140325104133",
    "deviceinfo.platform_version": "31.0a1",
    "deviceinfo.product_model": "ALCATEL ONE TOUCH FIRE",
    "deviceinfo.software": "Boot2Gecko 1.5.0.0-prerelease",
    "deviceinfo.update_channel": "default",
    "icc": {
        "mcc": "310",
        "mnc": "410",
        "spn": null
    },
    "locale": "en-US",
    "network": {
        "mcc": "310",
        "mnc": "410",
        "operator": "AT&T"
    },
    "pingID": "e426da9f-2a29-4e09-895b-c883903956cb",
    "pingTime": 1395852542588,
    "screenHeight": 480,
    "screenWidth": 320
}
</pre></code>


=Phase 1=
[https://wiki.mozilla.org/FirefoxOS/Metrics/FoxFooding Data gathered for the FoxFooding program]
==Requirements==
#Product Planning: Need to know the number of devices sold. This will help product decisions be data driven.
#User Benefit: knowing the following information will improve security updates and app functionality
## Build IDs, Country, Operator, OEM, Version of HW: These are needed for Device Updates and Security patches: When a bug is found, or a security risk is detected, they need to be sent as an update to the particular OEM, Operator.
##Screen Height, Width, Pixel density: This is needed for CoreApps development (Gallery, Camera, Dialer, Browser) and 3rd party apps. These apps need to know the screen sizes, resolutions that they would need to support.


==Design of Activation Ping==
=Virtual Team=
#Activation Ping is triggered through the first-time use experience / activation. Will automatically retry until it is successful (on a 1 hour interval)
*Device Engg:  Tamara Hills, Gregor Wagner, Russ Nicoletti
#Never pings again once it is successful (unless the phone's "userdata" partition is formatted through the recovery partition, but this is not a case we support generally, it means wiping all of the user's data)
*Device QA: Shing Lyu, Iris Hsiao, Gerry Chang
#Can also be disabled, but requires extra work on top of disabling app updates or removing the Firefox marketplace app
*Server Metrics: Dave Zeber, John Jensen
==Delta to what exists today==
*Planning: Dominik Strohmeier, Maria Oteo
#Information:
*Product Mgmt: Peter Dolanjski, Maria Oteo
##ActivationPing gives the total number of "sold"(activated) FFOS devices, whereas current ADI ping gives the number of devices connected to WiFi on a particular day
*Alumni: Marshall Culpepper, Hema Koka
###Product planning needs the information about devices sold (ActivatioPing). Current ADI ping approximated devices active on a certain day on WiFi. Difference gives us an idea of retention of product, and engagement of user.. This will help us focus on what is causing the dropoff, investigate why certain markets are doing better/worse, and making data-driven investments in improving the retention
##Activation Ping has additional information on Build IDs, Country, Operator, OEM, Version of HW, Screen Size, Screen Height, Width, Pixel density. While the previous ADI ping contains the model of device.
#Reliability:
##Current Ping from 1.3 shows approx 5 devices reporting as of March 24th 2014. https://dataviz.mozilla.org/views/ADI_Ping_1_3/HistoricalADITrend#2
#Cost:
##From previous versions, Partners removed ADI ping due to cost concern. Activation Ping is sent once in a lifetime, which should allay cost concerns.


=Phase 2=
=Links=
==Requirements==
*[https://wiki.mozilla.org/FirefoxOS/Metrics/CustomMetrics CustomMetrics]
*[https://wiki.mozilla.org/FirefoxOS/Metrics/EngineeringMetrics EngineeringMetrics]
*[https://wiki.mozilla.org/FirefoxOS/Metrics/activationping Activation Ping]
*[https://wiki.mozilla.org/FirefoxOS/Metrics/App_Usage AppUsage Data]
*[https://docs.google.com/a/mozilla.com/document/d/1i9CwaeLsOM-XS1Z8UZ88-b03CkPyxHI07sLhcZ9z4EY/edit#heading=h.lqni56rfu030 MAU Doc]
*[https://wiki.mozilla.org/FirefoxOS/Metrics Main Metrics Page (this page)]
*[https://wiki.mozilla.org/B2G/QA/Ping_Testing Testing Guide]
*[https://wiki.mozilla.org/Firefox_OS/Metrics/FAQ Frequently Asked Questions]
*[https://docs.google.com/document/d/1CxpSQzjvV-N3owFHpOWcki9dmNZa874-x2MJp4NxG-g/edit#heading=h.jgf7qkensd70 Performance Goals]

Latest revision as of 16:42, 21 January 2016

Goals

The FxOS Metrics project enables making data driven decisions for the following:

  • Understanding how FxOS is doing in the market versus product/business goals
  • Gaining insights from quantitative and qualitative data, to enable improve the product/business goals
  • Understanding usage models, to improve future product development and opportunities
  • Determining effectiveness of marketing campaigns.

Metrics

Customer lifecycle Metrics to be collected Dashboard
Acquiring Customers
  • Partners/Channels though which devices are sold
 Dashboard: Choose Operator/OEM/Country combination of channel

Activation of Devices

  • Total number of Activated devices
 Dashboard: Choose 'All' in SW/Operator/OEM/ to get a count

Retention Of Customers

  • MAU (Monthly Active Users)
  • Apps usage per month
  • Performance (App Startup time, Power consumption)
 Internal
Revenue
  • Number of searches done
  • Paid App Downloads
 Internal
Referral
  • Net Promoter Score
    • Recommendations from Users
    • User Satisfaction Factor
 Future

Roadmap

FxOS Release Metrics to be collected Bugs
1.3
  • Engg: 986701,997960
  • Legal: 980920
  • Privacy: 992487
2.1
  • Client: 982663
  • Legal: 1021259, 1109429
2.2
2.5
2.6 (May 2016) (In Planning Meta Bug for 2.6 (1226182))
End-2016 (In Planning)
  • Retention
    • Content Recommendation Engine based on usage (User Benefit)

User Stories

FoxFooding

Data gathered for the FoxFooding program

Virtual Team

  • Device Engg: Tamara Hills, Gregor Wagner, Russ Nicoletti
  • Device QA: Shing Lyu, Iris Hsiao, Gerry Chang
  • Server Metrics: Dave Zeber, John Jensen
  • Planning: Dominik Strohmeier, Maria Oteo
  • Product Mgmt: Peter Dolanjski, Maria Oteo
  • Alumni: Marshall Culpepper, Hema Koka

Links

Retrieved from "https://wiki.allizom.org/index.php?title=Firefox_OS/Metrics&oldid=1113701"