Firefox OS/Metrics: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Update the team)
 
(183 intermediate revisions by 8 users not shown)
Line 1: Line 1:
=FTU ping=
=Goals=
==Design of Activation Ping==
The FxOS Metrics project enables making data driven decisions for the following: 
#Activation Ping is triggered through the first-time use experience / activation.
*Understanding how FxOS is doing in the market versus product/business goals
#Never pings again once it is successful (unless the phone's "userdata" partition is formatted through the recovery partition, but this is not a case we support generally, it means wiping all of the user's data)
*Gaining insights from quantitative and qualitative data, to enable improve the product/business goals
#Can also be disabled, but requires extra work on top of disabling app updates or removing the Firefox marketplace app
*Understanding usage models, to improve future product development and opportunities
==Details==
*Determining effectiveness of marketing campaigns.
When a device is first activated, The FTU ping service sends a simple one-off HTTP request to our telemetry server. The HTTP request contains a JSON payload with some high level information about the device and network, but no unique hardware identifiers. A randomly generated pingID is sent to help with filtering on the server side, but this ID is discarded by the server once filtering has occurred (generally after 24 hours).


Currently, the logic for the FTU ping is broken down like this:
=Metrics=
# Unless the setting <tt>ftu.pingEnabled</tt> is explicitly set to false, start the FTU ping service. If the setting is missing, or any other value, then the ping service is enabled.
{| class="wikitable"
# Lazily get or create a randomly generated pingID, which is used for server side filtering. This pingID is only generated once per device, and is discarded on the server side.
|-
# Record the time the ping service started, to track the delta of time to a successful ping.
! Customer lifecycle!! Metrics to be collected !! Dashboard
# Pull all relevant ping payload settings asynchronously.
|-
# Create an interval timer to attempt a ping once an hour. This is configurable with <tt>ftu.pingTryInterval</tt>.
|Acquiring Customers
## Check for availability of voice network information. If none is available, try again on the next interval. If this fails 24 times, then continue trying the ping until is successful. This is configurable in the setting <tt>ftu.pingMaxNetworkFails</tt>.
|
## Send an HTTP request to <tt>FTU_PING_URL/PING_ID/ftu/FirefoxOS/VERSION/UPDATE_CHANNEL/BUILD_ID</tt>. By default, the ping URL is https://fxos.telemetry.mozilla.org, and is configurable in the <tt>ftu.pingURL</tt> setting, or via make with <tt>FTU_PING_URL</tt>.
*Partners/Channels though which devices are sold
### If the HTTP request succeeds, set <tt>ftu.pingEnabled</tt> to false, and disable the interval timer. Otherwise, try again on the next interval.
|[https://metrics.mozilla.com/protected/dashboards/fxos-ftu/ Dashboard]: Choose Operator/OEM/Country combination of channel
|-
|
Activation of Devices
|
*Total number of Activated devices
|[https://metrics.mozilla.com/protected/dashboards/fxos-ftu/ Dashboard]: Choose 'All' in SW/Operator/OEM/ to get a count
|-
|
Retention Of Customers
|
*MAU (Monthly Active Users)
*Apps usage per month
*Performance (App Startup time, Power consumption)
|Internal
|-
|Revenue
|
*Number of searches done
*Paid App Downloads
|Internal
|-
|Referral
|
*Net Promoter Score
**Recommendations from Users
**User Satisfaction Factor
|Future
|}


==Example payload and URL==
=Roadmap=
{| class="wikitable"
<tt>POST https://fxos.telemetry.mozilla.org/submit/telemetry/e426da9f-2a29-4e09-895b-c883903956cb/ftu/FirefoxOS/31.0a1/default/20140325104133</tt>
|-
!FxOS Release !! Metrics to be collected !! Bugs
|-
|1.3
|
*Acquisition, Activations:
**[https://wiki.mozilla.org/Firefox_OS/Metrics/activationping Activation Ping]:
|
*Engg: 986701,997960
*Legal: 980920
*Privacy: 992487
|-
|2.1
|
*Retention
**[https://wiki.mozilla.org/FirefoxOS/Metrics/App_Usage App Usage (Default Off)]
|
*Client: 982663
*Legal: 1021259, 1109429
|-
|2.2
|
*Retention
**[https://wiki.mozilla.org/FirefoxOS/Metrics/App_Usage App Usage (Default On)]
***[https://wiki.mozilla.org/FirefoxOS/Metrics/App_Usage AppUsage],
***[https://docs.google.com/a/mozilla.com/document/d/1i9CwaeLsOM-XS1Z8UZ88-b03CkPyxHI07sLhcZ9z4EY/edit#heading=h.lqni56rfu030 MAU]
*Acquisition:
**[https://bugzilla.mozilla.org/1021173 Report Preinstalled Apps]
*Revenue
**[https://bugzilla.mozilla.org/show_bug.cgi?id=1126524 Num of Searches/Revenue ]
|
*[https://wiki.mozilla.org/FirefoxOS/Metrics/App_Usage AppUsage], MAU:
**[https://bugzilla.mozilla.org/1109422 1109422 Client side work for enabling AppUsage ],
**[https://bugzilla.mozilla.org/1119009 1119009 Marketplace Apps Only]
**[https://bugzilla.mozilla.org/1109426 1109426 Server side dashboard for Most used apps: ]
*[https://bugzilla.mozilla.org/1021173 1021173 Report Preinstalled Apps, ]
*[https://bugzilla.mozilla.org/1126524 1126524 Search ]
|-
|2.5
|
*Retention:
**[https://bugzilla.mozilla.org/1179004 OTA collection of App Startup time (1179004)].
*Activation
**Accuracy of activation: [https://bugzilla.mozilla.org/1171685 FTU Ping: Avoid counting repeat pings from a known device. (1171685).]
*Privacy:
**[https://bugzilla.mozilla.org/1181295 Converged UX for opt-in, opt-out of metrics. 1181295.]
*Scaling:
**[https://bugzilla.mozilla.org/1152000 Framework (Metabug: 1152000) to collect custom Metrics and histogram]
**Non Phone devices support: [https://bugzilla.mozilla.org/1171685 Using TV serial number to avoid counting repeat pings. 1171685.]
**B2GDroid Support ([https://bugzilla.mozilla.org/show_bug.cgi?id=1198338 B2GDroid Metric Collection 1198338])
**AddOns Support ([https://bugzilla.mozilla.org/1198346: AddOns Metric collection 1198346])
|
|-
|2.6 (May 2016)
| (In Planning [https://bugzilla.mozilla.org/show_bug.cgi?id=1226182 Meta Bug for 2.6 (1226182)])
*Activation
**Optimizing FTU: % of signups to Moz services: FxAccounts, FMD
**Accuracy of Retention/MAU: [https://bugzilla.mozilla.org/1198164 FTU ping should include info on whether User opt-in for metrics (1198164)]
*Retention
**[https://bugzilla.mozilla.org/show_bug.cgi?id=1226458 Support for progressive web apps (1226458)]
**[https://bugzilla.mozilla.org/show_bug.cgi?id=1226274 Make uuid available to Foxfooder app (1226274)]
**Inclusion of device metrics in A/B testing (likely to be moved beyond 2.5)
**Engagement with device: [https://bugzilla.mozilla.org/1193225 Number of voice calls, data (wifi and cellular), number of text messages. 1193225.]
** Device Startup time: [https://bugzilla.mozilla.org/1177226 OTA collection of device dtartup time (1177226)]
**Advanced A/B testing
**Churn Rate & User Lifecycle Analysis (In planning)
**Framework Improvements
***Enum Type and Boolean Type histograms
**Support for New Security model [https://bugzilla.mozilla.org/show_bug.cgi?id=1225960 1225960]
**FoTA Ping [https://bugzilla.mozilla.org/show_bug.cgi?id=1202970 Engg bug: 1202970], [https://bugzilla.mozilla.org/show_bug.cgi?id=1202987. Legal bug: 1202987].
*Referral
**Net Promoter Score (User Satisfaction Factor)
*Privacy
**[https://bugzilla.mozilla.org/1126513 User ability to delete collected info. 1126513.]
*Server
**[https://bugzilla.mozilla.org/show_bug.cgi?id=1226062 Graphing of Addon Histograms]
**[https://bugzilla.mozilla.org/show_bug.cgi?id=1226064 Graphing of Keyed Histograms]
|-
|End-2016 (In Planning)
|
*Retention
**Content Recommendation Engine based on usage (User Benefit)
|
|}


<code><pre>
=User Stories=
{
*[https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AmVPziKCVEUEdGFoczVCaHBzOGxQcXNxWWs3aWRSalE#gid=0 User Stories]
    "activationTime": 1395769944966,
    "devicePixelRatio": 1,
    "deviceinfo.firmware_revision": "",
    "deviceinfo.hardware": "qcom",
    "deviceinfo.os": "1.5.0.0-prerelease",
    "deviceinfo.platform_build_id": "20140325104133",
    "deviceinfo.platform_version": "31.0a1",
    "deviceinfo.product_model": "ALCATEL ONE TOUCH FIRE",
    "deviceinfo.software": "Boot2Gecko 1.5.0.0-prerelease",
    "deviceinfo.update_channel": "default",
    "icc": {
        "mcc": "310",
        "mnc": "410",
        "spn": null
    },
    "locale": "en-US",
    "network": {
        "mcc": "310",
        "mnc": "410",
        "operator": "AT&T"
    },
    "pingID": "e426da9f-2a29-4e09-895b-c883903956cb",
    "pingTime": 1395852542588,
    "screenHeight": 480,
    "screenWidth": 320
}
</pre></code>


==Testing tips==
= FoxFooding =


Testing the FTU ping can be tedious and time consuming with the default settings. Here's a few things you can do to make it better:
[https://wiki.mozilla.org/FirefoxOS/Metrics/FoxFooding Data gathered for the FoxFooding program]


* When using build.sh or make, set FTU_PING_URL to a local IP so you can monitor the http request, i.e: <tt>cd gaia; make FTU_PING_URL=... reset-gaia</tt>
=Virtual Team=
* Set <tt>DEBUG = true</tt> in <tt>gaia/apps/system/js/ftu_ping.js</tt> to see messages at various stages of the ping lifecycle, and the raw ping data when it is actually sent
*Device Engg:  Tamara Hills, Gregor Wagner, Russ Nicoletti
* Change <tt>DEFAULT_TRY_INTERVAL</tt> in <tt>gaia/apps/system/js/ftu_ping.js</tt> to something much smaller than 1 hour (i.e. 15 seconds)
*Device QA: Shing Lyu, Iris Hsiao, Gerry Chang
*Server Metrics: Dave Zeber, John Jensen
*Planning: Dominik Strohmeier, Maria Oteo
*Product Mgmt: Peter Dolanjski, Maria Oteo
*Alumni: Marshall Culpepper, Hema Koka


=High Level Requirements=
=Links=
#Product Planning: Need to know the number of devices sold. This will help product decisions be data driven.
*[https://wiki.mozilla.org/FirefoxOS/Metrics/CustomMetrics CustomMetrics]
#User Benefit: knowing the following information will improve security updates and app functionality
*[https://wiki.mozilla.org/FirefoxOS/Metrics/EngineeringMetrics EngineeringMetrics]
## Build IDs, Country, Operator, OEM, Version of HW: These are needed for Device Updates and Security patches: When a bug is found, or a security risk is detected, they need to be sent as an update to the particular OEM, Operator.  
*[https://wiki.mozilla.org/FirefoxOS/Metrics/activationping Activation Ping]
##Screen Height, Width, Pixel density: This is needed for CoreApps development (Gallery, Camera, Dialer, Browser) and 3rd party apps. These apps need to know the screen sizes, resolutions that they would need to support.
*[https://wiki.mozilla.org/FirefoxOS/Metrics/App_Usage AppUsage Data]
 
*[https://docs.google.com/a/mozilla.com/document/d/1i9CwaeLsOM-XS1Z8UZ88-b03CkPyxHI07sLhcZ9z4EY/edit#heading=h.lqni56rfu030 MAU Doc]
=Delta to what exists today in Marketplace Ping=
*[https://wiki.mozilla.org/FirefoxOS/Metrics Main Metrics Page (this page)]
#Information:
*[https://wiki.mozilla.org/B2G/QA/Ping_Testing Testing Guide]
##ActivationPing gives the total number of "sold"(activated) FFOS devices, whereas current ADI ping gives the number of devices connected to WiFi on a particular day
*[https://wiki.mozilla.org/Firefox_OS/Metrics/FAQ Frequently Asked Questions]
###Product planning needs the information about devices sold (ActivatioPing). Current ADI ping approximated devices active on a certain day on WiFi. Difference gives us an idea of retention of product, and engagement of user.. This will help us focus on what is causing the dropoff, investigate why certain markets are doing better/worse, and making data-driven investments in improving the retention
*[https://docs.google.com/document/d/1CxpSQzjvV-N3owFHpOWcki9dmNZa874-x2MJp4NxG-g/edit#heading=h.jgf7qkensd70 Performance Goals]
##Activation Ping has additional information on Build IDs, Country, Operator, OEM, Version of HW, Screen Size, Screen Height, Width, Pixel density. While the previous ADI ping contains the model of device.  
#Reliability:
##Current Ping from 1.3 shows approx 5 devices reporting as of March 24th 2014. https://dataviz.mozilla.org/views/ADI_Ping_1_3/HistoricalADITrend#2
#Cost:
##From previous versions, Partners removed ADI ping due to cost concern. Activation Ping is sent once in a lifetime, which should allay cost concerns.

Latest revision as of 16:42, 21 January 2016

Goals

The FxOS Metrics project enables making data driven decisions for the following:

  • Understanding how FxOS is doing in the market versus product/business goals
  • Gaining insights from quantitative and qualitative data, to enable improve the product/business goals
  • Understanding usage models, to improve future product development and opportunities
  • Determining effectiveness of marketing campaigns.

Metrics

Customer lifecycle Metrics to be collected Dashboard
Acquiring Customers
  • Partners/Channels though which devices are sold
 Dashboard: Choose Operator/OEM/Country combination of channel

Activation of Devices

  • Total number of Activated devices
 Dashboard: Choose 'All' in SW/Operator/OEM/ to get a count

Retention Of Customers

  • MAU (Monthly Active Users)
  • Apps usage per month
  • Performance (App Startup time, Power consumption)
 Internal
Revenue
  • Number of searches done
  • Paid App Downloads
 Internal
Referral
  • Net Promoter Score
    • Recommendations from Users
    • User Satisfaction Factor
 Future

Roadmap

FxOS Release Metrics to be collected Bugs
1.3
  • Engg: 986701,997960
  • Legal: 980920
  • Privacy: 992487
2.1
  • Client: 982663
  • Legal: 1021259, 1109429
2.2
2.5
2.6 (May 2016) (In Planning Meta Bug for 2.6 (1226182))
End-2016 (In Planning)
  • Retention
    • Content Recommendation Engine based on usage (User Benefit)

User Stories

FoxFooding

Data gathered for the FoxFooding program

Virtual Team

  • Device Engg: Tamara Hills, Gregor Wagner, Russ Nicoletti
  • Device QA: Shing Lyu, Iris Hsiao, Gerry Chang
  • Server Metrics: Dave Zeber, John Jensen
  • Planning: Dominik Strohmeier, Maria Oteo
  • Product Mgmt: Peter Dolanjski, Maria Oteo
  • Alumni: Marshall Culpepper, Hema Koka

Links

Retrieved from "https://wiki.allizom.org/index.php?title=Firefox_OS/Metrics&oldid=1113701"