Firefox OS/Metrics: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Update the team)
 
(175 intermediate revisions by 8 users not shown)
Line 1: Line 1:
=FTU ping=
=Goals=
==Design of Activation Ping==
The FxOS Metrics project enables making data driven decisions for the following: 
#Activation Ping is triggered through the first-time use experience / activation.
*Understanding how FxOS is doing in the market versus product/business goals
#Never pings again once it is successful (unless the phone's "userdata" partition is formatted through the recovery partition, but this is not a case we support generally, it means wiping all of the user's data)
*Gaining insights from quantitative and qualitative data, to enable improve the product/business goals
#Can also be disabled, but requires extra work on top of disabling app updates or removing the Firefox marketplace app
*Understanding usage models, to improve future product development and opportunities
==Details==
*Determining effectiveness of marketing campaigns.
When a device is first activated, The FTU ping service sends a simple one-off HTTP request to our telemetry server. The HTTP request contains a JSON payload with some high level information about the device and network, but no unique hardware identifiers. A randomly generated pingID is sent to help with filtering on the server side, but this ID is discarded by the server once filtering has occurred (generally after 24 hours).


Currently, the logic for the FTU ping is broken down like this:
=Metrics=
# Unless the setting <tt>ftu.pingEnabled</tt> is explicitly set to false, start the FTU ping service. If the setting is missing, or any other value, then the ping service is enabled.
{| class="wikitable"
# Lazily get or create a randomly generated pingID, which is used for server side filtering. This pingID is only generated once per device, and is discarded on the server side.
# Record the time the ping service started, to track the delta of time to a successful ping.
# Pull all relevant ping payload settings asynchronously.
# Create an interval timer to attempt a ping once an hour. This is configurable with <tt>ftu.pingTryInterval</tt>.
## Check for availability of voice network information. If none is available, try again on the next interval. If this fails 24 times, then continue trying the ping until is successful. This is configurable in the setting <tt>ftu.pingMaxNetworkFails</tt>.
## Send an HTTP request to <tt>FTU_PING_URL/PING_ID/ftu/FirefoxOS/VERSION/UPDATE_CHANNEL/BUILD_ID</tt>. By default, the ping URL is https://fxos.telemetry.mozilla.org/submit/telemetry, and is configurable in the <tt>ftu.pingURL</tt> setting, or via make with <tt>FTU_PING_URL</tt>.
### If the HTTP request succeeds, set <tt>ftu.pingEnabled</tt> to false, and disable the interval timer. Otherwise, try again on the next interval.
 
==Example payload and URL==
<tt>POST https://fxos.telemetry.mozilla.org/submit/telemetry/e426da9f-2a29-4e09-895b-c883903956cb/ftu/FirefoxOS/31.0a1/default/20140325104133</tt>
 
<code><pre>
{
  "activationTime": 1398300000000,
  "pingTime": 1398300000000,
  "locale": "hu",
  "network": {
    "mcc": "216",
    "mnc": "70",
    "operator": "vodafone HU vodafone"
  },
  "icc": {
    "spn": "vodafone",
    "mcc": "216",
    "mnc": "70"
  },
  "info": {
    "appUpdateChannel": "default",
    "reason": "ftu",
    "appVersion": "31.0a1",
    "geoCountry": "HU",
    "appName": "FirefoxOS",
    "appBuildID": "20140423185429"
  },
  "deviceinfo.firmware_revision": "",
  "deviceinfo.hardware": "qcom",
  "deviceinfo.os": "2.0.0.0-prerelease",
  "deviceinfo.product_model": "GP-KEON",
  "deviceinfo.platform_version": "31.0a1",
  "deviceinfo.platform_build_id": "20140423185429",
  "deviceinfo.software": "Boot2Gecko 2.0.0.0-prerelease",
  "deviceinfo.update_channel": "default",
  "devicePixelRatio": 1,
  "screenWidth": 320,
  "screenHeight": 480,
  "ver": 2
}
</pre></code>
 
===Glossary===
 
{|
|-
|-
| <code>activationTime</code> || Timestamp indicating when the device was activated.
! Customer lifecycle!! Metrics to be collected !! Dashboard
|-
|-
| <code>pingTime</code> || Timestamp indicating when the FTU ping was sent. The ping is sent as soon as possible after activation, although it may be delayed if no network is available.
|Acquiring Customers
|
*Partners/Channels though which devices are sold
|[https://metrics.mozilla.com/protected/dashboards/fxos-ftu/ Dashboard]: Choose Operator/OEM/Country combination of channel
|-
|-
| <code>locale</code> || Platform localization.
|
Activation of Devices
|
*Total number of Activated devices
|[https://metrics.mozilla.com/protected/dashboards/fxos-ftu/ Dashboard]: Choose 'All' in SW/Operator/OEM/ to get a count
|-
|-
| <code>network</code> || Country and network information retrieved from the voice network connection at the time the ping is sent. Values should be null if no voice network is available. May be different from the SIM information if, eg.,  device is activated while roaming on another network.
|
Retention Of Customers
|  
*MAU (Monthly Active Users)
*Apps usage per month
*Performance (App Startup time, Power consumption)
|Internal
|-
|-
| <code>icc</code> || Country and network information retrieved from the SIM card at the time the ping is sent. Values should be null if no SIM is present.
|Revenue
|
*Number of searches done
*Paid App Downloads
|Internal
|-
|-
| <code>mcc</code> || Mobile country code.
|Referral
|-
|
| <code>mnc</code> || Mobile network code (identifies operator).
*Net Promoter Score
|-
**Recommendations from Users
| <code>operator</code> || Network operator name.
**User Satisfaction Factor
|-
|Future
| <code>spn</code> || SIM service provider name.
|}
|-
 
| <code>info</code> || Fields added server-side, read from submission URL.
=Roadmap=
|-
{| class="wikitable"
| <code>appUpdateChannel</code> || Read from <code>deviceinfo.update_channel</code>.
|-
| <code>appVersion</code> || Read from <code>deviceinfo.platform_version</code>.
|-
| <code>appName</code> || Name of platform generating ping (<code>FirefoxOS</code>).
|-
| <code>appBuildID</code> || Read from <code>deviceinfo.platform_build_id</code>.
|-
| <code>geoCountry</code> || Country from which the ping was sent. Added server-side using GeoIP lookup.
|-
| <code>reason</code> || A code identifying the type of ping. FTU is currently the only ping being sent.
|-
| <code>deviceinfo</code> || Fields read off the device.
|-
| <code>deviceinfo.firmware_revision</code> ||
|-
| <code>deviceinfo.hardware</code> || Chipset manufacturer.
|-
| <code>deviceinfo.os</code> || Gaia version.
|-
| <code>deviceinfo.product_model</code> || Device model name.
|-
| <code>deviceinfo.platform_version</code> || Gecko version.
|-
|-
| <code>deviceinfo.platform_build_id</code> || Gecko build ID. Generated from a timestamp - first 8 digits give date.
!FxOS Release !! Metrics to be collected !! Bugs
|-
|-
| <code>deviceinfo.software</code> || Name of Gaia distribution.
|1.3
|
*Acquisition, Activations:
**[https://wiki.mozilla.org/Firefox_OS/Metrics/activationping Activation Ping]:
|
*Engg: 986701,997960
*Legal: 980920
*Privacy: 992487
|-
|-
| <code>deviceinfo.update_channel</code> ||  
|2.1
|
*Retention
**[https://wiki.mozilla.org/FirefoxOS/Metrics/App_Usage App Usage (Default Off)]
|
*Client: 982663
*Legal: 1021259, 1109429
|-
|-
| <code>devicePixelRatio</code> ||  
|2.2
|
*Retention
**[https://wiki.mozilla.org/FirefoxOS/Metrics/App_Usage App Usage (Default On)]
***[https://wiki.mozilla.org/FirefoxOS/Metrics/App_Usage AppUsage],
***[https://docs.google.com/a/mozilla.com/document/d/1i9CwaeLsOM-XS1Z8UZ88-b03CkPyxHI07sLhcZ9z4EY/edit#heading=h.lqni56rfu030 MAU]
*Acquisition:
**[https://bugzilla.mozilla.org/1021173 Report Preinstalled Apps]
*Revenue
**[https://bugzilla.mozilla.org/show_bug.cgi?id=1126524 Num of Searches/Revenue ]
|
*[https://wiki.mozilla.org/FirefoxOS/Metrics/App_Usage AppUsage], MAU:
**[https://bugzilla.mozilla.org/1109422 1109422 Client side work for enabling AppUsage ],
**[https://bugzilla.mozilla.org/1119009 1119009 Marketplace Apps Only]
**[https://bugzilla.mozilla.org/1109426 1109426 Server side dashboard for Most used apps: ]
*[https://bugzilla.mozilla.org/1021173 1021173 Report Preinstalled Apps, ]
*[https://bugzilla.mozilla.org/1126524 1126524 Search ]
|-
|-
| <code>screenWidth</code> || Width of the screen in pixels.  
|2.5
|
*Retention:
**[https://bugzilla.mozilla.org/1179004 OTA collection of App Startup time (1179004)].
*Activation
**Accuracy of activation: [https://bugzilla.mozilla.org/1171685 FTU Ping: Avoid counting repeat pings from a known device. (1171685).]
*Privacy:
**[https://bugzilla.mozilla.org/1181295 Converged UX for opt-in, opt-out of metrics. 1181295.]
*Scaling:
**[https://bugzilla.mozilla.org/1152000 Framework (Metabug: 1152000) to collect custom Metrics and histogram]
**Non Phone devices support: [https://bugzilla.mozilla.org/1171685 Using TV serial number to avoid counting repeat pings. 1171685.]
**B2GDroid Support ([https://bugzilla.mozilla.org/show_bug.cgi?id=1198338 B2GDroid Metric Collection 1198338])
**AddOns Support ([https://bugzilla.mozilla.org/1198346: AddOns Metric collection 1198346])
|
|-
|-
| <code>screenHeight</code> || Height of the screen in pixels.  
|2.6 (May 2016)
| (In Planning [https://bugzilla.mozilla.org/show_bug.cgi?id=1226182 Meta Bug for 2.6 (1226182)])
*Activation
**Optimizing FTU: % of signups to Moz services: FxAccounts, FMD
**Accuracy of Retention/MAU: [https://bugzilla.mozilla.org/1198164 FTU ping should include info on whether User opt-in for metrics (1198164)]
*Retention
**[https://bugzilla.mozilla.org/show_bug.cgi?id=1226458 Support for progressive web apps (1226458)]
**[https://bugzilla.mozilla.org/show_bug.cgi?id=1226274 Make uuid available to Foxfooder app (1226274)]
**Inclusion of device metrics in A/B testing (likely to be moved beyond 2.5)
**Engagement with device: [https://bugzilla.mozilla.org/1193225 Number of voice calls, data (wifi and cellular), number of text messages. 1193225.]
** Device Startup time: [https://bugzilla.mozilla.org/1177226 OTA collection of device dtartup time (1177226)]
**Advanced A/B testing
**Churn Rate & User Lifecycle Analysis (In planning)
**Framework Improvements
***Enum Type and Boolean Type histograms
**Support for New Security model [https://bugzilla.mozilla.org/show_bug.cgi?id=1225960 1225960]
**FoTA Ping [https://bugzilla.mozilla.org/show_bug.cgi?id=1202970 Engg bug: 1202970], [https://bugzilla.mozilla.org/show_bug.cgi?id=1202987. Legal bug: 1202987].
*Referral
**Net Promoter Score (User Satisfaction Factor)
*Privacy
**[https://bugzilla.mozilla.org/1126513 User ability to delete collected info. 1126513.]
*Server
**[https://bugzilla.mozilla.org/show_bug.cgi?id=1226062 Graphing of Addon Histograms]
**[https://bugzilla.mozilla.org/show_bug.cgi?id=1226064 Graphing of Keyed Histograms]
|-
|-
| <code>ver</code> || Payload format version. Gets bumped when fields get added or changed.
|End-2016 (In Planning)
|
*Retention
**Content Recommendation Engine based on usage (User Benefit)
|
|}
|}


==Testing tips==
=User Stories=
*[https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AmVPziKCVEUEdGFoczVCaHBzOGxQcXNxWWs3aWRSalE#gid=0 User Stories]


Testing the FTU ping can be tedious and time consuming with the default settings. Here's a few things you can do to make it better:
= FoxFooding =


* When using build.sh or make, set FTU_PING_URL to a local IP so you can monitor the http request, i.e: <tt>cd gaia; make FTU_PING_URL=... reset-gaia</tt>
[https://wiki.mozilla.org/FirefoxOS/Metrics/FoxFooding Data gathered for the FoxFooding program]
* Set <tt>DEBUG = true</tt> in <tt>gaia/apps/system/js/ftu_ping.js</tt> to see messages at various stages of the ping lifecycle, and the raw ping data when it is actually sent
* Change <tt>DEFAULT_TRY_INTERVAL</tt> in <tt>gaia/apps/system/js/ftu_ping.js</tt> to something much smaller than 1 hour (i.e. 15 seconds)


=High Level Requirements=
=Virtual Team=
#Product Planning: Need to know the number of devices sold. This will help product decisions be data driven.
*Device Engg: Tamara Hills, Gregor Wagner, Russ Nicoletti
#User Benefit: knowing the following information will improve security updates and app functionality
*Device QA: Shing Lyu, Iris Hsiao, Gerry Chang
## Build IDs, Country, Operator, OEM, Version of HW: These are needed for Device Updates and Security patches: When a bug is found, or a security risk is detected, they need to be sent as an update to the particular OEM, Operator.
*Server Metrics: Dave Zeber, John Jensen
##Screen Height, Width, Pixel density: This is needed for CoreApps development (Gallery, Camera, Dialer, Browser) and 3rd party apps. These apps need to know the screen sizes, resolutions that they would need to support.
*Planning: Dominik Strohmeier, Maria Oteo
*Product Mgmt: Peter Dolanjski, Maria Oteo
*Alumni: Marshall Culpepper, Hema Koka


==Delta to what exists today in Marketplace Ping==
=Links=
#Information:
*[https://wiki.mozilla.org/FirefoxOS/Metrics/CustomMetrics CustomMetrics]
##ActivationPing gives the total number of "sold"(activated) FFOS devices, whereas current ADI ping gives the number of devices connected to WiFi on a particular day
*[https://wiki.mozilla.org/FirefoxOS/Metrics/EngineeringMetrics EngineeringMetrics]
###Product planning needs the information about devices sold (ActivatioPing). Current ADI ping approximated devices active on a certain day on WiFi. Difference gives us an idea of retention of product, and engagement of user.. This will help us focus on what is causing the dropoff, investigate why certain markets are doing better/worse, and making data-driven investments in improving the retention
*[https://wiki.mozilla.org/FirefoxOS/Metrics/activationping Activation Ping]
##Activation Ping has additional information on Build IDs, Country, Operator, OEM, Version of HW, Screen Size, Screen Height, Width, Pixel density. While the previous ADI ping contains the model of device.  
*[https://wiki.mozilla.org/FirefoxOS/Metrics/App_Usage AppUsage Data]
#Reliability:
*[https://docs.google.com/a/mozilla.com/document/d/1i9CwaeLsOM-XS1Z8UZ88-b03CkPyxHI07sLhcZ9z4EY/edit#heading=h.lqni56rfu030 MAU Doc]
##Current Ping from 1.3 shows approx 5 devices reporting as of March 24th 2014. https://dataviz.mozilla.org/views/ADI_Ping_1_3/HistoricalADITrend#2
*[https://wiki.mozilla.org/FirefoxOS/Metrics Main Metrics Page (this page)]
#Cost:
*[https://wiki.mozilla.org/B2G/QA/Ping_Testing Testing Guide]
##From previous versions, Partners removed ADI ping due to cost concern. Activation Ping is sent once in lifetime of device, which should have allay cost concerns.
*[https://wiki.mozilla.org/Firefox_OS/Metrics/FAQ Frequently Asked Questions]
*[https://docs.google.com/document/d/1CxpSQzjvV-N3owFHpOWcki9dmNZa874-x2MJp4NxG-g/edit#heading=h.jgf7qkensd70 Performance Goals]

Latest revision as of 16:42, 21 January 2016

Goals

The FxOS Metrics project enables making data driven decisions for the following:

  • Understanding how FxOS is doing in the market versus product/business goals
  • Gaining insights from quantitative and qualitative data, to enable improve the product/business goals
  • Understanding usage models, to improve future product development and opportunities
  • Determining effectiveness of marketing campaigns.

Metrics

Customer lifecycle Metrics to be collected Dashboard
Acquiring Customers
  • Partners/Channels though which devices are sold
 Dashboard: Choose Operator/OEM/Country combination of channel

Activation of Devices

  • Total number of Activated devices
 Dashboard: Choose 'All' in SW/Operator/OEM/ to get a count

Retention Of Customers

  • MAU (Monthly Active Users)
  • Apps usage per month
  • Performance (App Startup time, Power consumption)
 Internal
Revenue
  • Number of searches done
  • Paid App Downloads
 Internal
Referral
  • Net Promoter Score
    • Recommendations from Users
    • User Satisfaction Factor
 Future

Roadmap

FxOS Release Metrics to be collected Bugs
1.3
  • Engg: 986701,997960
  • Legal: 980920
  • Privacy: 992487
2.1
  • Client: 982663
  • Legal: 1021259, 1109429
2.2
2.5
2.6 (May 2016) (In Planning Meta Bug for 2.6 (1226182))
End-2016 (In Planning)
  • Retention
    • Content Recommendation Engine based on usage (User Benefit)

User Stories

FoxFooding

Data gathered for the FoxFooding program

Virtual Team

  • Device Engg: Tamara Hills, Gregor Wagner, Russ Nicoletti
  • Device QA: Shing Lyu, Iris Hsiao, Gerry Chang
  • Server Metrics: Dave Zeber, John Jensen
  • Planning: Dominik Strohmeier, Maria Oteo
  • Product Mgmt: Peter Dolanjski, Maria Oteo
  • Alumni: Marshall Culpepper, Hema Koka

Links

Retrieved from "https://wiki.allizom.org/index.php?title=Firefox_OS/Metrics&oldid=1113701"