Program Management/Programs/Telemetry/Improve Opt-in: Difference between revisions
(Created page with "=Goal= The current Telemetry opt-in rate is roughly 2%. The rate needs to improve to 10%+ in order for the data to be meaningful. want to get 10% plus on release channel dev (ni...") |
|||
| (35 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
=Goal= | =Goal= | ||
It is thought that the current Telemetry opt-in rates do not provide enough data to be useful. In general, Telemetry is only useful if users are providing feedback. The issue with Telemetry data lies in the sample population. Because Telemetry is opt-in, the sample population is not random and therefore needs to be very large (80-90%) to answer a broad range of questions. It may still be possible to answer specific questions with a smaller sample size (metrics can engage on an ongoing basis to assist with determination of whether a question can be answered). Alternatively, the existing sample population may contain enough variety to be representative. | |||
===Release and Beta channels=== | |||
Current: ~2%<br/> | |||
Goal: | |||
- | |||
===Nightly (M-C) and Aurora channels=== | |||
Current: ~2% <br/> | |||
Goal: ~100%<br/> | |||
<br/> | |||
Nightly and Aurora have small user bases. It is important to have as close to full adoption as possible in order to obtain meaningful data. Note that the users on these channels are assumed to be more technically savvy and interested in contributing to the improvement of Firefox. | |||
==Investigation== | |||
The metrics team (Chris Jung) will investigate to see if the current Telemetry data is representative based on the information that we have available from daily add-on pings. See {{bug|698848}}. | |||
=Tactics= | =Tactics= | ||
The following are some tactics that may be used to increase the opt-in rate for Telemetry. | The following are some tactics that may be used to increase the opt-in rate for Telemetry. | ||
==Change to opt-out policy== | ==Change to enabled by default (opt-out policy) on Aurora and Nightly== | ||
Will change to enabled by default (opt-out policy) on Aurora and Nightly channels. See {{bug|699806}} Enable Telemetry by default on Nightly and Aurora channels. | |||
<bugzilla> | |||
{ | |||
"id": "699806,702284,728340,728350,701182,723129,725407,725987,733522,735444,737596,701178,819732" | |||
} | |||
</bugzilla> | |||
==Fix current desktop opt-in ui== | |||
Issues with the current opt-in UI: | |||
* UI may disappear quickly | |||
* UI may be hidden in another tab | |||
These issues are related to the prompt mechanism (used for know your rights message) not being designed to be used heavily. | |||
<bugzilla> | |||
{ | |||
"id": "691951,697860,737600" | |||
} | |||
</bugzilla> | |||
==Improve Telemetry opt-in advertising== | |||
The current Telemetry notification has not provided the opt-in rates that we would like to see to gather meaningful information from the Firefox population. We should look for alternative ways to advertise Telemetry. We can look to use the Mozilla blog (others?) to promote Telemetry and request user help with improving the performance of Firefox. | |||
<bugzilla> | |||
{ | |||
"id": "692861,683340,731433" | |||
} | |||
</bugzilla> | |||
== | ==Enable Telemetry on mobile== | ||
Telemetry needs to be added to the mobile native UI. | |||
<bugzilla> | |||
{ | |||
"id": "696076,699513,702281" | |||
} | |||
</bugzilla> | |||
Latest revision as of 15:44, 10 December 2012
Goal
It is thought that the current Telemetry opt-in rates do not provide enough data to be useful. In general, Telemetry is only useful if users are providing feedback. The issue with Telemetry data lies in the sample population. Because Telemetry is opt-in, the sample population is not random and therefore needs to be very large (80-90%) to answer a broad range of questions. It may still be possible to answer specific questions with a smaller sample size (metrics can engage on an ongoing basis to assist with determination of whether a question can be answered). Alternatively, the existing sample population may contain enough variety to be representative.
Release and Beta channels
Current: ~2%
Goal:
Nightly (M-C) and Aurora channels
Current: ~2%
Goal: ~100%
Nightly and Aurora have small user bases. It is important to have as close to full adoption as possible in order to obtain meaningful data. Note that the users on these channels are assumed to be more technically savvy and interested in contributing to the improvement of Firefox.
Investigation
The metrics team (Chris Jung) will investigate to see if the current Telemetry data is representative based on the information that we have available from daily add-on pings. See bug 698848.
Tactics
The following are some tactics that may be used to increase the opt-in rate for Telemetry.
Change to enabled by default (opt-out policy) on Aurora and Nightly
Will change to enabled by default (opt-out policy) on Aurora and Nightly channels. See bug 699806 Enable Telemetry by default on Nightly and Aurora channels.
13 Total; 0 Open (0%); 12 Resolved (92.31%); 1 Verified (7.69%);
Fix current desktop opt-in ui
Issues with the current opt-in UI:
- UI may disappear quickly
- UI may be hidden in another tab
These issues are related to the prompt mechanism (used for know your rights message) not being designed to be used heavily.
| ID | Summary | Priority | Status |
|---|---|---|---|
| 691951 | Telemetry prompt gets dismissed without user interaction | -- | RESOLVED |
| 697860 | Hide close button in the telemetry notification, and track explicit clicks on "no" | -- | VERIFIED |
| 737600 | When telemetry is disabled in the pref pane, we should update toolkit.telemetry.rejected | -- | RESOLVED |
3 Total; 0 Open (0%); 2 Resolved (66.67%); 1 Verified (33.33%);
Improve Telemetry opt-in advertising
The current Telemetry notification has not provided the opt-in rates that we would like to see to gather meaningful information from the Firefox population. We should look for alternative ways to advertise Telemetry. We can look to use the Mozilla blog (others?) to promote Telemetry and request user help with improving the performance of Firefox.
| ID | Summary | Priority | Status |
|---|---|---|---|
| 683340 | Modal Dialog box for opting in to Telemetry | -- | RESOLVED |
| 692861 | Design a more effective telemetry opt-in | -- | RESOLVED |
| 731433 | Prompt users who didn't opt in/out originally to opt into telemetry (beta channel) | -- | RESOLVED |
3 Total; 0 Open (0%); 3 Resolved (100%); 0 Verified (0%);
Enable Telemetry on mobile
Telemetry needs to be added to the mobile native UI.
| ID | Summary | Priority | Status |
|---|---|---|---|
| 696076 | Telemetry for Mobile | P4 | VERIFIED |
| 699513 | Advertise Telemetry opt-in on mobile | P2 | VERIFIED |
| 702281 | Telemetry prompt on mobile should be tri-state | P5 | VERIFIED |
3 Total; 0 Open (0%); 0 Resolved (0%); 3 Verified (100%);