Auto-tools/Projects/peptest

Overview

Peptest is an automated testing framework designed to test whether or not the browser's UI thread remains responsive while performing a variety of actions. Tests are simple Javascript files which can optionally import Mozmill's driver to manipulate the user interface in an automated fashion.

The code is currently located at https://github.com/mozilla/peptest. If you would like to help you can send pull requests, or ping ahal on irc for ways to get started.

For a longer description of what 'unresponsiveness' actually means, see: http://blog.mozilla.com/ted/2011/06/27/measuring-ui-responsiveness/

Using Peptest

Running Tests

Currently tests are run from the command line with python. Peptest currently depends on some external Mozilla python packages, namely: mozrunner, mozprocess, mozprofile, mozinfo, mozlog and manifestdestiny. These packages all live in the mozbase repository.

Clone the mozbase repository at github.com/mozilla/mozbase.
Create a new virtualenv and install the mozinfo, mozlog, mozprofile, mozprocess, mozrunner and manifestdestiny packages into it (run 'python setup.py install' for each or run 'python setup_development.py' to install all the packages).
Clone the Peptest repo at github.com/mozilla/peptest and install peptest (python setup.py install)

Run the command (use --help for a full list of options)

peptest --binary <path_to_binary> --test-path <path_to_test_manifest>

Test Manifest

All parameters are optional except for the --test-path parameter. This parameter should contain the path to a manifest that lists all the tests that will be run. Manifests should be of the following format:

# test paths are relative to the manifest
[test1.js]
[test2.js]
[foo/bar/test3.js]
# this test is disabled and won't be run
[test4.js]
disabled = "Doesn't work, see bug 123456"

Alternatively, the --test-path parameter can point to a single test which might be useful for debugging purposes.

Peptest uses ManifestDestiny. See additional documentation.

Fail Threshold

One more thing that can be specified from the manifest is the failure threshold. When each test finishes, the Peptest harness will aggregate all of the unresponsive times into one number that can be used for regression detection (see metric). If you specify a failThreshold for a test, it will only be reported as a failure if it's metric is higher than the threshold.

# no failThreshold, this test will fail if there is any unresponsiveness whatsoever
[test1.js]
# these tests will only fail if their metrics are greater than their failThresholds
[test2.js]
failThreshold = 42
[test3.js]
failThreshold = 500

Test Format

Tests are simply Javascript files that will be executed in chrome space. This means they have access to any of the API's that an extension would normally have access to. In addition to this, they also have access to Mozmill's driver for convenient automation of Firefox/Thunderbird's UI.

It is helpful to think of tests as a series of actions. That is, the UI thread will only be checked for responsiveness while an action is currently happening. This ensures that we are only testing the actions that we care about, and that the test isn't overrun with noise generated during setup or teardown. To accomplish this, tests call a function called 'performAction(name, func)' which takes in two parameters, a name and a function pointer whose contents consist of a single action.

The following example test will make sure that the browser remains responsive while opening a page and while searching in Google.

// import mozmill and initialize a controller
Components.utils.import('resource://mozmill/driver/mozmill.js');
let controller = getBrowserController();

// Create our first action which will open Google
performAction('open_google', function() {
  controller.open('http://google.com');
  controller.waitForPageLoad();
});

// stuff not inside a performAction() call won't be tested for responsiveness
let textbox = findElement.ID(controller.tabs.activeTab, 'lst-ib');
let button = findElement.Name(controller.tabs.activeTab, 'btnK');

// Create our second action which will perform a search in the google searchbox
performAction('search_google', function() {
  textbox.sendKeys('foobar');
  button.click();
  controller.waitForPageLoad();
});

For documentation on using Mozmill's driver see: The Mozmill Reference Desk

Note that using mozmill may be convenient but is not required. Here is another example that tests resizing the window.

// getWindow is a convenience function exposed to tests
// It uses nsIWindowMediator to get the current window
let window = getWindow();
let width = window.outerWidth;
let height = window.outerHeight;

performAction('resize_by', function() {
  window.resizeBy(100, 100);
});

performAction('resize_to', function() {
  window.resizeTo(800, 600);
});

// Tests should clean up after themselves
window.resizeTo(width, height);

There are a few example Peptests in the repo.

Log Format

Peptest uses the mozlog module to standardize on the logging format.


PEP TEST-START | test1.js
PEP TEST-PASS  | test1.js | fail threshold: 0.0 | metric: 0.0
PEP TEST-END   | test1.js | finished in: 1536 ms
PEP TEST-START | test2.js
PEP WARNING    | test2.js | loadPage | unresponsive time: 116 ms
PEP WARNING    | test2.js | loadPage | unresponsive time: 61 ms
PEP WARNING    | test2.js | loadPage | unresponsive time: 275 ms
PEP TEST-UNEXPECTED-FAIL | test2.js | fail threshold: 90.0 | metric: 92.802
PEP TEST-END   | test2.js | finished in: 8022 ms
PEP TEST-START | test3.js
PEP WARNING    | test3.js | scroll_menu | unresponsive time: 121 ms
PEP TEST-PASS  | test3.js | fail threshold: 15.0 | metric: 14.641
PEP TEST-END   | test3.js | finished in: 2137 ms

When a responsiveness measurement runs over 50ms a TEST-UNEXPECTED-FAIL message is output. The second column is the name of the test file, the third column is the name of the particular action that was being performed and the last column is the time in milliseconds that the event took to process.

When the test is finished, the total run time and a metric are output. The metric is explained in the next section.

Metric

The metric is a number that is calculated based on all accumulated unresponsive times. The function used to calculate the metric is:

# values is a list of unresponsive values that occurred during the test
def get_metric(values):
    return sum([x*x / 1000.0 for x in values])

The metric is useful for determining whether a test has regressed or not. For example if a test used to have a metric of 56.43 then all of a sudden has a metric of 193.42 we know a regression has happened.

Important: You cannot compare the metric from one test to that of another. Tests can be of an arbitrary length and therefore it is impossible to conclude that a test with a metric of 10 is more responsive than a test with a metric of 1000. You must only compare metric values to previous metric values of the same test.

Further Work

Mozmill e10s

Note: this will likely be put on hold indefinitely until the status of e10s becomes more clear

Mozmill doesn't use MessageManager when it interacts with content. This means that it will not work correctly with Electrolysis builds (which is mostly the entire point of Peptests). Mozmill will need to be refactored to use MessageManager and also to stop using the gBrowser object which doesn't exist in mobile Firefox.

Detailed results

Dietrich has written some tools to determine exactly which Javascript calls were unresponsive. This is a very useful tool which should be incorporated into the Peptest harness. More details can be found here

I (ahal) would really like someone to own bug 580055, un-bitrot the patch and get it checked in (with ifdefs or something). If you want, I'll even un-bitrot the patch for you :)

Buildbot Integration

Work has been done on writing a mozharness peptest script in bug 692091. There is a mozharness script checked in and work is being done to get it running on tryserver.

Related Bugs

bug 580055 - responsiveness profiling (blocked on bug 702740)
bug 674606 - initial harness creation
~~bug 692091~~ - mozharness script
bug 700415 - peptest on tryserver
~~bug 700656~~ - peptest in m-c/packaged tests
bug 703266 - mirror mozbase to m-c
bug 703269 - mirror peptest harness to m-c

API Reference

There is a Peptest API that gets injected into the scope of every test under the 'pep' namespace. Each test can use the following methods:

General

pep.performAction(name, function)

Performs an action during which responsiveness measurements are recorded

name - the name of the action to run
function - a function pointer of the action to run

pep.getWindow(type)

Returns a reference to the current window

type - string representing the window type, defaults to "navigator:browser" (see windowType)

pep.sleep(milliseconds)

Unfortunately UI Automation is very dependent on timing and sometimes it is necessary to sleep, though you should avoid this method if you can

milliseconds - the number of milliseconds to sleep for

Logging

For logging messages to terminal

msg - the message to log

Logging methods

Auto-tools/Projects/peptest

Contents

Overview

Using Peptest

Running Tests

Test Manifest

Fail Threshold

Test Format

Log Format

Metric

Further Work

Mozmill e10s

Detailed results

Buildbot Integration

Related Bugs

API Reference

General

pep.performAction(name, function)

pep.getWindow(type)

pep.sleep(milliseconds)

Logging

pep.log.debug(msg)

pep.log.info(msg)

pep.log.warning(msg)

pep.log.error(msg)

Navigation menu

Auto-tools/Projects/peptest

Overview

Using Peptest

Running Tests

Test Manifest

Fail Threshold

Test Format

Log Format

Metric

Further Work

Mozmill e10s

Detailed results

Buildbot Integration

Related Bugs

API Reference

General

pep.performAction(name, function)

pep.getWindow(type)

pep.sleep(milliseconds)

Logging

pep.log.debug(msg)

pep.log.info(msg)

pep.log.warning(msg)

pep.log.error(msg)

Navigation menu

Search