View source for QA/Telemetry/Developing a Telemetry Experiment

{{outdated}}

= Telemetry Experiments =

This page describes the mechanics of how to build, test, and deploy a Telemetry Experiment. For a more high-level overview of Telemetry Experiments, see [[Telemetry/Experiments]].

== Getting started ==

* clone the repo http://hg.mozilla.org/webtools/telemetry-experiment-server/
* create a new folder under experiments/ for your new experiment
** it is highly recommended that you use `hg cp` from an existing folder (this makes review easier as it's able to track the changes made to the boilerplate files)
** e.g. 
*** <code>cd experiments; hg cp e10s-beta45-withoutaddons my-new-experiment</code>
** use a descriptive folder name to highlight the feature, channel and version
* the main experiment code lives in code/bootstrap.js

== Manifest file ==

* unlike other restartless extensions, experiments must have a top-level manifest.json file, at experiments/your-experiment/manifest.json.
* this file contains the configuration details for your experiment - you can use manifests for other experiments as examples
* the "sample" value, for example, tells the percentage of users (matching the other criteria) that will receive your experiment
* Note that the name field is visible to end users, so it should be accurate and not confusing.

== install.rdf file ==

* the file under experiments/your-experiment/code/install.rdf file contains details about the experiment add-on. The <type> value for all experiments should be 128. Note that an add-on with type=128 does not work by drag'n'dropping it to Firefox like a regular add-on (it needs to be loaded by the experiment system)

== Building the experiment ==

* To build the experiment, you just need to create an .xpi file with your bootstrap.js and install.rdf files. Put this xpi in the root of your experiment folder (i.e., experiments/your-experiment/)
* If you grab one [http://hg.mozilla.org/webtools/telemetry-experiment-server/file/tip/experiments/e10s-beta45-withoutaddons/code/Makefile example Makefile] from the tree, you just need to use `make experiment` and the xpi will be created for you
** note that the xpi is created under a build/ folder. You need to move it the proper location (the root of your experiment)

== Building the server ==

* To test, you need to build the telemetry server (which is a simple static folder). To build it, you need to setup a virtualenv (see the README.rst file in the root) and then run in the root of the repo:
** <code>python build.py ../build http://localhost:8080</code>
* you can change ../build to any folder where you want to output it, '''but note:''': this folder will be erased by the build.py script
* the last argument should be a string with the base URL of the server that you're going to set up  (it's used to fill some data in the generated .json file).  If you plan to publish the test server in a different path (e.g., people.mozilla.org), you will need to change it
* If you get an error while building it, you have something to fix in your code.
* Now you need to run a http server rooted at that folder. You can do that easily with python with:
** <code>cd ../build</code>
** <code>python -m SimpleHTTPServer 8080</code>
* Now it's a good opportunity to check if everything worked. Visit http://localhost:8080 if your normal web browser to see if a page shows up and it has the correct information, similar to [http://telemetry-experiment.cdn.mozilla.net/ the production page].
* Verify that the information about your experiment is correct.
* Also verify that the generated firefox-manifest.json is there at http://localhost:8080/firefox-manifest.json

== Testing everything ==

* '''Very important''': before testing, you need to make sure that telemetry is enabled (which it isn't for a local build). On a local build, you can use <code>export MOZ_TELEMETRY_REPORTING=1</code> in your mozconfig, or set '''toolkit.telemetry.enabled''' to true.
* '''Also important''', if you built locally, your update channel will be ''default'', which probably won't match what you specified in your manifest. Add ''default'' to your experiment manifest or test with a correct nightly/aurora/beta build.

To test things, you need to set up a number of prefs:

** '''devtools.chrome.enabled = true''', in order for the Browser Console to show up
** '''toolkit.telemetry.enabled = true''', telemetry must be enabled
** '''experiments.manifest.uri = "http://localhost:8080/firefox-manifest.json"''', full path to your generated firefox-manifest.json
** '''experiments.force-sample-value = "0.1"''', to make sure your testing always falls into the sample group
** '''experiments.logging.enabled = true''', for logging
** '''experiments.logging.level = 0''', more logging
** '''extensions.logging.enabled = true''', even more logging from the Add-ons Manager
** '''xpinstall.signatures.required = false''', to allow installing unsigned add-ons

Once you enable '''devtools.chrome.enabled''' from about:config, you can paste the rest into the Browser console and be happy:

<code><pre>
Services.prefs.setBoolPref("xpinstall.signatures.required", false);
Services.prefs.setBoolPref("toolkit.telemetry.enabled", true);
Services.prefs.setBoolPref("experiments.logging.enabled", true);
Services.prefs.setBoolPref("extensions.logging.enabled", true);
Services.prefs.setIntPref("experiments.logging.level", 0);
Services.prefs.setCharPref("experiments.force-sample-value", "0.1");
Services.prefs.setCharPref("experiments.manifest.uri", "http://localhost:8080/firefox-manifest.json");
</pre></code>

== Forcing a manifest update ==
Firefox updates the manifest on a timer (and when you change the pref), but to force an update immediatelly, you should go to the Browser Console and do the following:

<code><pre>
Cu.import("resource:///modules/experiments/Experiments.jsm");
Experiments.instance().updateManifest();
</pre></code>

This has the great advantage that you'll see all the logging right in front of you too in the Browser console.

== Verifying that things are working ==

** Go to the Add-ons manager (Ctrl/Cmd + A, or about:addons), to see if there's a new section, "Experiments", and your experiment is there and active.
** Go to about:support, scroll to the very bottom, and see if your experiment is listed there and verify that it's in the correct branch that you expected.
** Check prefs and see if your experiment had its intended behavior

== Re-testing ==

After you've already tested things once, you might be wondering how to clean everything up to re-test everything. You should do:
** In the Add-ons Manager, go to your Experiment and uninstall it
** '''And''' go to your profile folder and remove the file '''experiments.json'''.

This will make Firefox forget everything about your experiment. Now you can use the snippet above (under the "Forcing a manifest update" section), to start everything again.

If you changed the manifest and rebuilt the server, you may need to navigate to the firefox-manifest.json file in the browser and force a refresh (Ctrl/Cmd + Shift + R), for it to pick new changes

== Testing on Beta ==

If your experiment is going to run on beta or release, and you want to test on those channels, you'll '''need''' to have the XPI signed (see below), or rebuild beta to remove the requirement that experiments be signed, by [https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/extensions/internal/XPIProvider.jsm?rev=002d7897c93c#267 removing this line]. It's probably better to use one of the following two workflows:

* test on a local beta build modified to allow unsigned add-ons, then get code review, then sign the XPI, and then deploy to staging
* get code review, sign the XPI, test the signed XPI on a stock beta build, and then deploy to staging

These workflows put the signing step after code review, because the reviewer might request changes to the add-on which will require you to re-sign the updated version. Neither of these workflows are mandatory, though - do what makes sense.

== Code Review ==

Before deploying to staging, you should get your experiment reviewed. To request reviews, file a bug against the [https://bugzilla.mozilla.org/enter_bug.cgi?product=Firefox%20Health%20Report&component=Client%3A%20Desktop Firefox Health Report product, Desktop Client component].

== Signing the XPI ==

For testing on a stock Beta build, or for deployment, you'll need to have the experiment signed. You can file a bug against the addons.mozilla.org product, Addon Validation component, attach the xpi file, and request that it be signed. Ping Jason Thomas if you run into problems.

== Deploying to staging ==

Once everything looks good, everything is working perfectly, you've got r+, and the experiment signed, you should push it to the repo (this requires [https://www.mozilla.org/en-US/about/governance/policies/commit/access-policy/ commit access level 3]).

Pushing to the repo will build the server to the [https://telemetry-experiment-dev.allizom.org/ staging server]. You can then use the [https://telemetry-experiment-dev.allizom.org/firefox-manifest.json firefox-manifest.json] from the staging server to test your experiment.

At this point, you can also request QA testing of your experiment if needed. Refer to the [[Telemetry/Experiments#QA_Testing|QA Testing]] section of the main experiment documentation for details.

== Deploying to production ==

Before deploying, you need to send an e-mail to release-drivers with an intent to ship notice, and get approval from release mgmt. See the [[Telemetry/Experiments#Final_Deployment|Final Deployment]] section of the main experiment documentation for the email template.

To deploy, you need to update the ''release_tag'' in the repo to point to your newest changeset:
* <code> hg tag -f --rev tip release_tag -m "Bug AAA - Update release_tag to deploy my experiment. r=someone"</code>
Then, file a bug ([https://bugzilla.mozilla.org/show_bug.cgi?id=1247822 example]) for DevOps to deploy it to the [https://telemetry-experiment.cdn.mozilla.net/ production server].