General

Test Pilot experiments are javascript modules which are hosted like web content on https://testpilot.mozillalabs.com. They are downloaded by the Test Pilot extension and are run with full chrome privileges. They have access to XPCOM and can do everything that extensions do. With great power comes great responsibility!

A Test Pilot experiment has a data store (held locally on the user's machine in SQLite), it has some observers which watch for events and write them into the data store, and it has a little bit of HTML web content which is displayed to the user in order to explain what the experiment is doing.

The extension runs your experiment for a certain period of time that you specify. When that period is over, the user will be asked to click a "submit" button to upload their data. According to our privacy policy, no data is uploaded except when the user explicitly gives this permission.

The Test Pilot extension manages the lifecycle of each experiment, as well as providing all the user interface and handling the data uploads. You don't have to worry about any of that stuff in writing an extension. You just need to specify the observers that you want to install, the schema of the data store, the web content, and a little bit of metadata.

Aside from experiments, you can also run surveys with Test Pilot. The API for defining a survey is much simpler than that for an experiment; it is mainly just a JSON object with some text for each question and for each answer in the case of multiple-choice questions. A survey can be attatched to a study or it can be standalone.

Test Pilot experiments and surveys are implemented as Javascript Securable Modules. They will be loaded using Cuddlefish. They must implement the required API by exporting objects with certain specific names that the Test Pilot extension expects to see. The main function of this page is to document that API, what each object must be named and what it can do.

Test pilot experiments and surveys are hosted on https://testpilot.mozillalabs.com. This URL is hard-coded into the Test Pilot extension. Note that the experiments and surveys are served exclusively over SSL to reduce the possibility that a man-in-the-middle could substitute malicious code.

The Index files

The server holds an index file which lists experiments to run. The clients check this file (on startup, then once per day thereafter) and then download the experiments they find listed there. To release a new experiment or survey, we add it to the index file. To stop giving out an experiment or survey, we take it out of the index file. (Clients who already have it will keep running it).

The canonical index file is at https://testpilot.mozillalabs.com/testcases/index.json. You can go have a look at it right now. It's human readable and fairly self-explanatory.

The top level of the JSON has a property called experiments, which is an array of objects with names and filenames:

{"experiments": [{"name": "Tab Open/Close Study",
                 "filename": "tab-open-close/tabs_experiment.js"},
                {"name": "Survey For New Test Pilots",
                 "filename": "new_pilot_survey.js"}]}

In order to develop and test experiments that we're not ready to deploy, we have an alternate index file for developers only. This lives at https://testpilot.mozillalabs.com/testcases/index-dev.json. To make your copy of the Test Pilot client connect to this file instead of the public/canonical one, use the following steps:

Go to about:config
Find the pref called extensions.testpilot.indexFileName
Change its value to index-dev.json
Delete your locally cached copy of index.json, if any. If present, it will be at (your profile directory)/TestPilotExperimentFiles/index.json
Restart Firefox

Once you make that switch, you will start getting all the studies that are still in development, as well as some dummy studies which exist for testing purposes only. If you want to develop your own study, you should start using index-dev.json.

Version control for experiment/survey code

Experiment and survey code lives in the Testpilot website Hg repository: http://hg.mozilla.org/labs/testpilotweb .

Specifically, index.json is in the testcases directory within this Hg repository. Since our website is just a checkout of this repo, that puts the file at a URL of https://testpilot.mozillalabs.com/testcases/index.json . The paths and URLs of all survey/extension code are relative to index.json.

Working with Cuddlefish

See https://wiki.mozilla.org/Labs/Jetpack/JEP/28 for detailed information about using Cuddlefish.

Since your experiment code is a Cuddlefish securable module, it has access to the following globals...

To import another securable module:

var module = require( moduleName );

To export functions, constants, variables, or objects:

exports.foo = foo;

To provide destructor/cleanup code to be run when your module is unloaded:

require("unload").when(function myDestructor() {
    // do stuff here
  });

To log informational or error messages:

console.debug("Timestamped an event.");
console.info("Something happened.");
console.warn("Uh-oh: something happened.");
console.error("Something bad happened!!");

You have access to XPCOM through the following defined constants:

Cc is mapped to Components.classes.
Ci is mapped to Components.interfaces.
Cu is mapped to Components.utils.
Cr is mapped to Components.results.

Surveys

Surveys are very simple to write. You merely need one js module file that exports an object called surveyInfo. The surveyInfo object must have the following properties:

surveyInfo.surveyId: a unique string that identifies this survey. Can be anything but must be distinct from the ids of any other surveys. Will be used as part of a Mozilla preference so it must not contain spaces.
surveyInfo.surveyName: a string containing the human-readable name of the survey, which will be displayed in the extension's menus, etc.
surveyInfo.summary: A couple sentences to be displayed in the Test Pilot interface explaining what the survey is about.
surveyInfo.thumbnail: A URL to an image file that will be displayed (at 90x90 pixels) next to the summary.
surveyInfo.versionNumber: Required. The version number of the survey. Increment this number when you change the survey questions, so that when you're looking at survey results you can tell which version they came from.
surveyInfo.minTPVersion: A version string (e.g. "1.0a1") corresponding to a version of the Test Pilot extension. Versions of the extension older than this will not attempt to download the survey. If not specified, all versions of the extension will download.
surveyInfo.uploadWithExperiment: Optional. If this survey goes with another study, then set this to the ID of that study to ensure that the survey results and study results get uploaded together and can be correlated.
surveyInfo.surveyQuestions: Required. An array of question objects. See the example file below to see how to write question objects.
surveyInfo.onPageLoad: Optional function that can execute arbitrary javascript (with chrome privileges) when the survey page loads.

The presence or absence of the surveyInfo object is used by the extension to determine whether a JS module is a survey or an experiment. Any JS module that exports an object named surveyInfo will be treated as a survey. If you are writing an experiment, make sure not to export an object named surveyInfo!

Example

There is an example survey in the hg checkout, at

testcases/example/example_survey.js

You can also see it on the web right now at https://testpilot.mozillalabs.com/testcases/example/example_survey.js. This survey is pretty silly but it demonstrates the json format for defining survey questions and possible answers.

Experiments

Experiment API definition

Experiments are much more involved than surveys, and must export (using the Cuddlefish export() function) four objects, named exactly as follows:

experimentInfo
dataStoreInfo
webContent
handlers

The interfaces and function of these objects are explained one by one in the following sections.

Base Classes

There are base classes provided in the file study_base_classes.js which implement experimentInfo, dataStoreInfo, webContent, and handlers for you. Instead of implementing these objects from scratch, the easiest way to write an experiment is to import the base classes into your file (using Cuddlefish:)

var BaseClasses = require("study_base_classes.js");

and then extend the base classes, overriding whatever methods you need to customize. This method is strongly recommended.

Example Experiment and Tutorial

There is an example experiment file in the hg checkout at testcases/example/example_study.js. You can also see the code for it on the web right now by going to https://testpilot.mozillalabs.com/testcases/example/example_study.js.

The example experiment works by extending the Base Classes. It is heavily commented and intended to be as self-explanatory as possible. The best way to get started writing an experiment is to copy that file and start changing stuff.

There is a Test Pilot Experiment Tutorial that walks you through the steps of creating an experiment using the example experiment and the base classes.

Documentation of Experiment API By Object

experimentInfo

This object contains metadata about the experiment: its name and id, its version number, when it is to be run and for how long, whether it recurs, whether it requires an additional layer of opt-in, and the URLs for further information about it. It should contain the following fields:

startDate: if null, then the experiment will start automatically as soon as it is installed. If you want to instead schedule a date in the future for the test to start, instead set it to a parseable date string.
duration: an integer, giving the number of days the test is to run for. E.g. if duration is 14 then the test will run from startDate to startDate + 2 weeks. Optional; duration defaults to 7 days if not specified.
testName: The human readable name of the experiment, as it will be displayed in menus and status pages, etc. (The word "Study" will be appended by the client, so the testName should not also have the word "Study" in it or the end result will look silly.) Required.
testId: A unique number, used by the client to track the progress of multiple experiments running at once. An experiment must not have the same testId as any other experiment or survey. Required.
testInfoUrl: the URL of a page where the user can go to get more information about the experiment.
optInRequired: Set this to true if the test should not start automatically on the start date, but should instead only start if the user explicitly tells it to start. This should be the case for tests that change the Firefox UI, such as A-B tests. Note that the UI to support optInRequired has not yet been implemented in the client as of version 1.0.
recursAutomatically: Set this to true if the test should recur automatically after it ends.
recurrenceInterval: If recursAutomatically is true, then this is the number of days (measured from the startDate) after which the study will begin again. For obvious reasons, this must be longer than duration. Has no effect if recursAutomatically is false.
versionNumber: An integer for telling apart different versions of the same experiment. Whenever a significant change is made to the experiment code, the versionNumber should be incremented. The version number gets attached to the user's data upload when they submit results for the experiment; it can then be detected on the server side to tell apart submissions from different versions of the experiment. Required.
minTPVersion: A version string (e.g. "1.0a1") corresponding to a version of the Test Pilot extension. Versions of the extension older than minTPVersion will not attempt to download or run this study. It is optional; if not specified, all versions of the extension will download and run the study.
minFXVersion: A version string (e.g. "4.0b3") corresponding to a version of Firefox. Test Pilot running on a version of Firefox older than minFXVersion will not attempt to download or run this study. If not specified, the study will be run on all versions of Firefox.

experimentInfo Example

The experimentInfo object for the Tab open/close study is as follows:

exports.experimentInfo = {
  startDate: null,
  duration: 7,
  testName: "Tab Open/Close Study",
  testId: 1,
  testInfoUrl: "https://testpilot.mozillalabs.com/testcases/tab-open-close.html",
  testResultsUrl: "https://testpilot.mozillalabs.com/testcases/tab-open-close/results.html",
  optInRequired: false,
  recursAutomatically: false,
  recurrenceInterval: 0,
  versionNumber: 2
};

dataStoreInfo

dataStoreInfo provides parameters for the initialization of a client-side SQLite database table which will be used to store the events that your experiment records. It's basically a schema description, that is fed to an object-relational mapper built into the extension.

When the Test Pilot extension runs your experiment, it will generate a Store object based on the specification you gave it in dataStoreInfo. Your observers (see below) will be able to call a method to record objects into this store. The info you provide in dataStoreInfo determines how the properties of this event object get mapped to the columns in the database.

dataStoreInfo must have the following properties (all required):

fileName: The name of the file where the SQLite database should be kept (inside the user's profile directory). It should be as descriptive as possible and end with ".sqlite".
tableName: The name of the database table to create inside this file. It should be descriptive and must not have spaces.
columns: An array of objects, each describing a single database column. Each column corresponds to one property on the event objects that you plan to pass in to storeEvent(). The properties that each column object must specify are as follows:
- property: What property of the event object will be mapped to this column. For example, if you plan to pass in event objects that have a .timestamp property, then you will want a column object with property: "timestamp".
- type: A code specifying the data storage type of the column. Supported types are TYPE_INT_32, TYPE_DOUBLE, and TYPE_STRING which are all defined in study_base_classes.js.
- displayName: The human-readable name that will be used to label this column whenever we display database contents. Can have spaces. Should be Title Cased.
- displayValue (optional): If not provided, then the raw value that is stored will be what is displayed whenever we display database contents. If it is provided, it tells the client how to display the contents of this column. It must be either an array of strings, or a function that returns a string. If it's a function, the raw value will be passed into the function and the return value will be what's displayed. If it's an array, the raw value will be used as an index into the array, and the string at that index will be what's displayed (useful for enumerated type codes).
- default (not yet implemented): What value to use if the eventObject passed in does not have a property matching this column.

Example of dataStoreInfo

exports.dataStoreInfo = {
  fileName: "testpilot_tabs_experiment_results.sqlite",
  tableName: "testpilot_tabs_experiment",
  columns: [
    {property: "event_code", type: TYPE_INT_32, displayName: "Event",
     displayValue: ["", "Open", "Close", "Drag", "Drop", "Switch", "Load",
                    "Startup","Shutdown", "Window Open", "Window Close"]},
    {property: "tab_position", type: TYPE_INT_32, displayName: "Tab Pos."},
    {property: "tab_window", type: TYPE_INT_32, displayName: "Window ID"},
    {property: "ui_method", type: TYPE_INT_32, displayName: "UI Method",
     displayValue: ["", "Click", "Keyboard", "Menu", "Link", "URL Entry", 
                    "Search", "Bookmark", "History"]},
    {property: "tab_site_hash", type: TYPE_INT_32, displayName: "Tab Group ID"},
    {property: "num_tabs", type: TYPE_INT_32, displayName: "Num. Tabs"},
    {property: "timestamp", type: TYPE_DOUBLE, displayName: "Time",
     displayValue: function(value) {return new Date(value).toLocaleString();}}
 ]};

Example of storing event objects

In observer code:

this.record({
  event_code: TabsExperimentConstants.DRAG_EVENT,
  timestamp: Date.now(),
  tab_position: index,
  num_tabs: event.target.parentNode.itemCount,
  ui_method: TabsExperimentConstants.UI_CLICK,
  tab_window: windowId
});

This anonymous event object passed to storeEvent has properties matching the properties of the columns specified in dataStoreInfo; it provides all the expected properties except for tab_site_hash, which will therefore default to 0.

webContent

The webContent object is used to fill in the content of the experiment status page, which lives on the client side as part of the extension (it's shown to the user at url chrome://testpilot/content/status.html). webContent must expose a certain list of properties, documented below. Instead of implementing all of these from scratch, you should probably extend the base webContent class in study_base_classes.js and override only the properites you want to customize.

Each of these properties is a string (or a getter that returns a string) containing a chunk of HTML. The HTML chunks appropriate to the experiment's current status will be selected by logic in the extension, and slotted into place on the experiment status page.

upcomingHtml: Shown before the experiment begins.
inProgressHtml: Shown while experiment is in progress.
canceledHtml: Shown if the experiment was canceled by the user.
completedHtml: Shown after the experiment is complete, before it is submitted.
remainDataHtml: Shown after experiment data submitted but before it expires.
dataExpiredHtml: Shown if data expired without being submitted.
deletedRemainDataHtml: Shown if data expired after being submitted.
inProgressDataPrivacyHtml: Shown in privacy info sidebar while experiment is in progress.
canceledDataPrivacyHtml: Shown in privacy info sidebar if experiment was canceled by the user.
completedDataPrivacyHtml: Shown in privacy info sidebar after experiment is complete, before it is submitted.
remainDataPrivacyHtml: Shown in privacy info sidebar after experiment data is submitted, before it expires.
dataExpiredDataPrivacyHtml: Shown in privacy info sidebar if data expired without being submitted.
deletedRemainDataPrivacyHtml: Shown in privacy info sidebar if data expired after being submitted.

webContent must also provide the following function:

onPageLoad: A function, that will be called when the status page has loaded. This can do whatever the experiment needs to do to finish rendering the page. For example, it can pull data from the data store and draw graphs on canvas elements.

The onPageLoad function will be called with the following arguments:

onPageLoad(experiment, document, graphUtils);

The values of these arguments are as follows:

experiment: reference the 'live' experiment object itself. This reference can be used to get the contents of the data store (as CSV or as JSON), to check the experiment status, etc. etc. TODO: document public API of experiment object
document: the status page document, the one you are running in. (Because this code is in a securable-module context, the document reference would not otherwise be available.) You can use the document object to do "document.getElementById()" and get references to tags in your html and do fun things.
graphUtils: A reference to a live instance of the flot JS chart-plotting library, which you can use to draw graphs as you see fit.

handlers

The handlers object exported by your experiment has the very important responsibility of actually doing the data collection, and inserting the collected data into the store you defined in dataStoreInfo.

There is only a single handlers object; it is a global singleton and not, for example, a per-window object.

handlers must define the following functions, named exactly as they appear here. Each function is called by the extension when the appropriate event occurs. Exactly what your handlers do in response to these events is up to you.

onNewWindow(window): Called when a new window is opened, with a reference to the window object.
onWindowClosed(window): Called when a window is closed, with a reference to the window object.
onAppStartup(): Called when Firefox starts up.
onAppShutdown(): Called when Firefox shuts down.
onExperimentStartup(store): Called when your experiment is starting up, with a reference to the data store object.
onExperimentShutdown(): Called when your experiment is to be shut down.
doExperimentCleanup(): Called when your experiment is completed or canceled by the user and will not be run anymore.
onEnterPrivateBrowsing(): Called when the user enters Private Browsing Mode.
onExitPrivateBrowsing(): Called when the user exits Private Browsing Mode.

To respect the user's privacy, your handlers should not record any data while the user is in Private Browsing Mode.

"Experiment Startup" is called when your experiment has just started running for the first time. It's also called when the user starts up Firefox again after exiting, in which case your study will need to resume running. In the latter case, your onExperimentStartup() method will be called immediately followed by the onAppStartup() method.

Most experiments will do the majority of their work (registering listeners on various UI elements) in the onExperimentStartup() method or the onNewWindow() method, depending on whether the observation is global or per-window.

In your onExperimentStartup() method, you should save a reference to the data store object since you will need to use its storeEvent() method to store data.

The file study_base_classes.js defines a class called GenericGlobalObserver that defines appropriate responses to each of these events. It also defines a GenericWindowObserver helper class for installing per-window observers, which is a very common use case. I strongly recommend extending GenericGlobalObserver, overriding methods you want to customize, and then setting exports.handlers to a new instance of your subclass. This is explained in more detail in the /Labs/Test_Pilot/Experiment_Tutorial Experiment Tutorial.

Publishing an Experiment or Survey

If you have implemented an experiment or survey and want to distribute it to Test Pilot users, please consult the Test Pilot team for code review and scheduling. There is no guarantee that we will run your study as we can only run studies that meet our security and privacy standards; decisions will be made on a case-by-case basis.

Labs/Test Pilot/ExperimentAPI

Contents