Labs/Test Pilot/debug

Information about getting the Test Pilot source code, installing the development version, where the important code lives, seeing debugging output, useful tricks for debugging and testing changes, etc.

To install Test Pilot extension on Firefox:

The most recent release of the extension is at https://addons.mozilla.org/services/install.php?addon_id=testpilot

However, we really want to be trying to reproduce and fix bugs in the development trunk, not the latest released version. To install the version from the development trunk:

1. Make sure you have Python and Mercurial installed. 2. hg clone http://hg.mozilla.org/labs/testpilot/ 3. cd testpilot 4. python manage.py install <profile> (where <profile> is the name of the Firefox profile to install the extension to; you may want to create a new profile for this purpose first.)

The Code: Most of the important stuff is in testpilot/extension/modules:

testpilot/extension/modules/tasks.js - defines the base class and subclasses for "Tasks". Tasks can be surveys or experiments. Most of the Test Pilot extension code is concerned with managing Tasks: tracking their state, providing a UI for the user to look at them all, etc.

 testpilot/extension/modules/setup.js - the core module, contains the

logic for managing the list of tasks, also handles UI elements such as the task bar menu and popup notifications.

 testpilot/extension/modules/remote-experiment-loader.js - downloads JS

securable modules from testpilot.mozillalabs.com and loads them. (experiment logic and survey questions are implemented in these securable modules; after being loaded by remote-experiment-loader, they are passed back to setup.js, which wraps them in new Task instances and manages them.) Remote-experiment-loader uses the "Cuddlefish" library, an implementation of the JS Securable Module standard, so for full understanding it will help to get familiar with the documentation here: https://wiki.mozilla.org/Labs/Jetpack/JEP/25

 testpilot/extension/modules/experiment_data_store.js - persistence

layer; provides interface for each experiment to create/read/update/delete its own table in the built-in Firefox SQLite database.

 testpilot/extension/content/browser.js - startup code, some menu item

handlers

 testpilot/extension/components/TestPilot.js - startup code

testpilot/extension/content/status.html - Page that gets displayed (through chrome URL) to user to show current status of an experiment and allow user to interact with that experiment by looking at charts, clicking Submit, or opting out. This is the main UI for Test Pilot.

testpilot/extension/content/experiment-page.js - Javascript code that populates status.html with relevant information. The entry point is the function loadExperimentPage(). The code that handles clicks on the Upload button is also here.

Configuring the Test Pilot directory index: Test Pilot uses a special directory file to tell it what tests to download and run. The default directory file is at https://testpilot.mozillalabs.com/testcases/index.json

There is a different directory file just for developers and testers called index-dev.json. This is so we can tell the extension to run experiment code that we are in the process of testing, before we release that extension code to the wider audience. So, you'll want to sign up for index-dev.json. To do this:

1. Open about:config 2. search for "testpilot" 3. Find the pref "extensions.testpilot.indexFileName" 4. Change the value to "index-dev.json".

Debugging output:

The Test Pilot extension emits a ton of logging and debugging output. It goes to standard out, so the way to see it is to start Firefox up from the command line in a terminal, and then watch that terminal for the test pilot output.

(Note: I know this is messy. I intend to replace this with a proper system of sending debugging output to a log file.)

Manually changing the state of an experiment: This is very useful for testing, because otherwise you would have to wait for a week to pass before a test was ready to submit! The status of each task (experiments or surveys) is stored in preferences. You can manually edit these through about:config.

There are two relevant preferences for each one: extensions.testpilot.startDate.(id) and extensions.testpilot.taskstatus.(id). So for example, the ID of the menu item usage study is 4, so its starting date is stored in extensions.testpilot.startdate.4, and its status is in extensions.testpilot.taskstatus.4.

The taskstatus is one of the following codes (defined in modules/tasks.js):

0: New 1: Pending 2: Starting 3: In progress 4: Finished 5: Cancelled 6: Submitted 7: Results 8: Archived

If a task is currently in progress and you want to fast-forward to the end (e.g. to test submission of a study), just change its taskstatus preference to 4 (finished).

If a task has already ended and you want to reset it so that you can run it again from the beginning, you can just right-click (or ctrl-click) on its taskstatus preference and its startDate preference and "Reset" both of them.

ID codes for all experiments and surveys that have been run so far: 1 = Tab study 2 = Week-in-the-life study 3 = Accounts and Passwords study 4 = Menu item study 1000 = "Ghost study" (this study doesn't do anything; I'm just using it for purposes of testing and debugging some server-side stuff.)

Surveys have strings instead of numerical ID codes. (There was a reason for this difference once, but the reason is no longer valid, so it's just for backwards compatibility.) The two surveys that have been defined are: "account_password_survey" and "basic_panel_survey".

The Debug Page

When Test Pilot is running, enter chrome://testpilot/content/debug.html into the location bar to get to the debug page.

There are no links to this page nor any way to load it from the UI, because users are never supposed to see it (I made it just for my own use), but there are tons of useful things you can do here to debug Test Pilot problems. I use it constantly when developing Test Pilot. You may find it helpful too.

You can enter the ID number for an experiment (e.g. for the Menu Item Usage study, ID number = 4) into the field on top that says "Experiment number". Then...

  * Click "Show Raw Data" to see current contents of the database table

for this experiment.

  * Click "Wipe my data" to delete everything from that database table.
  * Click "Popup" to test the popup notification (showing status of

selected study).

  * Click "Reload all experiments" to do that without having to restart

Firefox

  * Click "Run Unit Tests"... oh yeah, I used to have unit tests |:-P  

They have not been maintained lately. I need to get on that. This button will not work.

  * Click "Remind me" to test popup notification (showing reminder of

pending tasks)

  * Enter a numerical status field in the next field and click "Set" to

set the status of the experiment to that code (I usually do this through about:config instead.)

  * Click "Show Metadata" to see the metadata that will be attatched to

the header of the uploaded data, including firefox version, locale, operating system, installed extensions, etc. to verify correctness.

  * Click "Load Ye Code" to load up the JS code implementing an

experiment (note that the experiment loaded is hard-coded into the source of debug.html, so you have to edit that and reload the page in order to get a different experiment). You can then edit the code right in the box (or do what I do, edit it in Emacs and then copy-paste it into this box) and click "Save and Run ye code" to test out changes to the experiment code. This is a shortcut I made so that I could rapidly test changes without having to upload new code to the server and restart Test Pilot every time I wanted to change something.

If you have more questions not answered here, please contact jono@mozilla.com. I will give your questions highest priority and answer them as soon as possible.