|
|
| (27 intermediate revisions by one other user not shown) |
| Line 1: |
Line 1: |
| = General =
| | {{warning|The original Mozilla Labs Test Pilot project has been retired. Click the 'View History' link on this page to read the old content.}} |
|
| |
|
| Test Pilot experiments and surveys are implemented as Javascript Securable Modules. They will be loaded using Cuddlefish. They are expected to export objects with certain specific names. | | As of January 2016 a similar initiative, formerly named Idea Town, has been [http://micropipes.com/blog/2016/01/27/meet-the-new-test-pilot/ named Test Pilot]. |
|
| |
|
| Test pilot experiments and surveys are hosted on https://testpilot.mozillalabs.com. This URL is hard-coded into the Test Pilot extension. Note that the experiments and surveys are served exclusively over SSL to reduce the possibility that a man-in-the-middle could substitute malicious code.
| | '''[[Test_Pilot|Learn more about the new Test Pilot!]]''' |
| | |
| == index.json ==
| |
| | |
| TODO
| |
| | |
| == The Hg repository for experiment/survey code ==
| |
| | |
| TODO
| |
| | |
| == Working with Cuddlefish ==
| |
| | |
| TODO
| |
| | |
| = Surveys =
| |
| | |
| Surveys are very simple to write, because we currently use SurveyMonkey to host the actual questions, meaning that the JS module on testpilot.mozillalabs.com only has to provide metadata. The survey js file simply has to export an object named surveyInfo which has the following properties:
| |
| | |
| * surveyInfo.surveyId: a unique string that identifies this survey. Can be anything but must be distinct from the ids of any other surveys. Will be used as part of a Mozilla preference so it must not contain spaces.
| |
| * surveyInfo.surveyName: a string containing the human-readable name of the survey, which will be displayed in the extension's menus, etc.
| |
| * surveyInfo.surveyUrl: the URL where the survey is hosted - users who choose to take the survey will be sent here.
| |
| * surveyInfo.resultsUrl: the URL where the survey results are hosted - users who choose to see survey results will be sent here. If there are no survey results yet (e.g. if it's a new survey), then resultsUrl should be undefined (i.e. don't specify it at all).
| |
| | |
| The presence or absence of the surveyInfo object is used by the extension to determine whether a JS module is a survey or an experiment. Any JS module that exports an object named surveyInfo will be treated as a survey. If you are writing an experiment, make sure not to export an object named surveyInfo!
| |
| | |
| == Example ==
| |
| | |
| new_pilot_survey.js, the file providing the metadata for "Survey for New Test Pilots", is reproduced here in its entirety:
| |
| | |
| exports.surveyInfo = {
| |
| surveyId: "survey_for_new_pilots",
| |
| surveyName: "Survey For New Test Pilots",
| |
| surveyUrl: "http://www.surveymonkey.com/s.aspx?sm=bxR0HNhByEBfugh8GPASvQ_3d_3d",
| |
| resultsUrl: "http://www.surveymonkey.com/sr.aspx?sm=oZPWdDCVgnJqkmPERROH6AWWPcmTImSDiMyFunw16b8_3d"
| |
| };
| |
| | |
| = Experiments =
| |
| | |
| Experiments are much more involved than surveys, and must export four objects, named ''exactly'' as follows:
| |
| | |
| # experimentInfo
| |
| # dataStoreInfo
| |
| # webContent
| |
| # Observer
| |
| | |
| These are explained one by one in the following sections.
| |
| | |
| == experimentInfo ==
| |
| | |
| This object contains metadata about the experiment: its name and id, its version number, when it is to be run and for how long, whether it recurs, whether it requires an additional layer of opt-in, and the URLs for further information about it. It should contain the following fields:
| |
| | |
| * startDate: if null, then the experiment will start automatically as soon as it is installed. If you want to instead schedule a date in the future for the test to start, instead set it to a parseable date string.
| |
| * duration: an integer, giving the number of days the test is to run for. E.g. if duration is 14 then the test will run from startDate to startDate + 2 weeks. Optional; duration defaults to 7 days if not specified.
| |
| * testName: The human readable name of the experiment, as it will be displayed in menus and status pages, etc. Required.
| |
| * testId: Can be either a number or a string, but must be unique - an experiment must not have the same id as another experiment or survey. These IDs will end up used in preference names, so it must not contain spaces. Required.
| |
| * testInfoUrl: the URL of a page where the user can go to get more information about the experiment.
| |
| * testResultsUrl: the URL of a page where experiment results have been published. If no results are published yet, leave this undefined. The presence of a defined testResultsUrl property is what lets the Test Pilot extension know that a test has moved on into the results phase, and update its UI accordingly.
| |
| * optInRequired: Set this to true if the test should not start automatically on the start date, but should instead only start if the user explicitly tells it to start. This should be the case for tests that change the Firefox UI, such as A-B tests. Note that the UI to support optInRequired has not yet been implemented in the client as of version 0.3.
| |
| * recursAutomatically: Set this to true if the test should recur automatically after it ends. Note that support for recurrence has not yet been implemented in the client as of version 0.3.
| |
| * recurrenceInterval: If recursAutomatically is true, then this is the number of days (measured from the startDate) after which the study will begin again. For obvious reasons, this must be longer than duration. Has no effect if recursAutomatically is false.
| |
| * versionNumber: An integer for telling apart different versions of the same experiment. Whenever a significant change is made to the experiment code, the versionNumber should be incremented. The version number gets attached to the user's data upload when they submit results for the experiment; it can then be detected on the server side to tell apart submissions from different versions of the experiment. Required.
| |
| | |
| | |
| === Example ===
| |
| | |
| The experimentInfo object for the Tab open/close study is as follows:
| |
| | |
| exports.experimentInfo = {
| |
| startDate: null,
| |
| duration: 7,
| |
| testName: "Tab Open/Close Study",
| |
| testId: 1,
| |
| testInfoUrl: "https://testpilot.mozillalabs.com/testcases/tab-open-close.html",
| |
| testResultsUrl: "https://testpilot.mozillalabs.com/testcases/tab-open-close/results.html",
| |
| optInRequired: false,
| |
| recursAutomatically: false,
| |
| recurrenceInterval: 0,
| |
| versionNumber: 2
| |
| };
| |
| | |
| == dataStoreInfo ==
| |
| | |
| dataStoreInfo provides parameters for the initialization of a client-side SQLite database table which will be used to store the events that your experiment records. It's basically a schema description, that is fed to an object-relational mapper built into the extension.
| |
| | |
| Your Observer objects (see below) will get passed a Store object, and whenever they want to record an event, they'll pass an event object into Store.storeEvent();. The info you provide in dataStoreInfo determines how the properties of this event object get mapped to the columns in the database.
| |
| | |
| dataStoreInfo must have the following properties (all required):
| |
| | |
| * fileName: The name of the file where the SQLite database should be kept (inside the user's profile directory). It should be as descriptive as possible and end with ".sqlite".
| |
| * tableName: The name of the database table to create inside this file. It should be descriptive and must not have spaces.
| |
| * columns: An array of objects, each describing a single database column. Each column corresponds to one property on the event objects that you plan to pass in to storeEvent(). The properties that each column object must specify are as follows:
| |
| ** property: What property of the event object will be mapped to this column. For example, if you plan to pass in event objects that have a .timestamp property, then you will want a column object with property: "timestamp".
| |
| ** type: A code specifying the data storage type of the column. Not all data types are supported by the client yet. (TODO: explain this better)
| |
| ** displayName: The human-readable name that will be used to label this column whenever we display database contents. Can have spaces. Should be Title Cased.
| |
| ** displayValue (optional): If not provided, then the raw value that is stored will be what is displayed whenever we display database contents. If it is provided, it tells the client how to display the contents of this column. It must be either an array of strings, or a function that returns a string. If it's a function, the raw value will be passed into the function and the return value will be what's displayed. If it's an array, the raw value will be used as an index into the array, and the string at that index will be what's displayed (useful for enumerated type codes).
| |
| ** default (not yet implemented): What value to use if the eventObject passed in does not have a property matching this column.
| |
| | |
| === Example of declaration ===
| |
| | |
| const TABS_EXPERIMENT_FILE = "testpilot_tabs_experiment_results.sqlite";
| |
| /* In this schema, each row represents a single UI event. */
| |
| | |
| const TABS_TABLE_NAME = "testpilot_tabs_experiment";
| |
| | |
| exports.dataStoreInfo = {
| |
| fileName: TABS_EXPERIMENT_FILE,
| |
| tableName: TABS_TABLE_NAME,
| |
| columns: TABS_EXPERIMENT_COLUMNS
| |
| };
| |
| {property: "event_code", type: TYPE_INT_32, displayName: "Event",
| |
| displayValue: ["", "Open", "Close", "Drag", "Drop", "Switch", "Load", "Startup",
| |
| "Shutdown", "Window Open", "Window Close"]},
| |
| {property: "tab_position", type: TYPE_INT_32, displayName: "Tab Pos."},
| |
| {property: "tab_window", type: TYPE_INT_32, displayName: "Window ID"},
| |
| {property: "ui_method", type: TYPE_INT_32, displayName: "UI Method",
| |
| displayValue: ["", "Click", "Keyboard", "Menu", "Link", "URL Entry", "Search",
| |
| "Bookmark", "History"]},
| |
| {property: "tab_site_hash", type: TYPE_INT_32, displayName: "Tab Group ID"},
| |
| {property: "num_tabs", type: TYPE_INT_32, displayName: "Num. Tabs"},
| |
| {property: "timestamp", type: TYPE_DOUBLE, displayName: "Time",
| |
| displayValue: function(value) {return new Date(value).toLocaleString();}}];
| |
| | |
| | |
| | |
| === Example of storing event objects ===
| |
| | |
| == webContent ==
| |
| | |
| == Observer ==
| |
Warning: The original Mozilla Labs Test Pilot project has been retired. Click the 'View History' link on this page to read the old content.