Auto-tools/Projects/peptest

From MozillaWiki
< Auto-tools‎ | Projects
Revision as of 15:49, 21 November 2011 by Ahal (talk | contribs)
Jump to navigation Jump to search

Overview

Peptest is an automated testing framework designed to test whether or not the browser's UI thread remains responsive while performing a variety of actions. Tests are simple Javascript files which can optionally import Mozmill's driver to manipulate the user interface in an automated fashion.

The code is currently located at https://github.com/mozilla/peptest. If you would like to help you can send pull requests, or ping ahal on irc for ways to get started.

Using Peptest

Running Tests

Currently tests are run from the command line with python. Peptest currently depends on some external Mozilla python packages, namely: mozrunner, mozprocess, mozprofile, mozinfo, mozlog and manifestdestiny. These packages all live in the mozbase repository.

  1. Clone the mozbase repository at github.com/mozilla/mozbase.
  2. Create a new virtualenv and install the mozinfo, mozlog, mozprofile, mozprocess, mozrunner and manifestdestiny packages into it (run 'python setup.py install' for each or run 'python setup_development.py' to install all the packages).
  3. Clone the Peptest repo at github.com/mozilla/peptest
  4. Run the command (use --help for a full list of commands)
    python runpeptests.py --binary <path_to_binary> --test-path <path_to_test_manifest>

Test Manifest

All parameters are optional except for the --test-path parameter. This parameter should contain the path to a manifest that lists all the tests that will be run. Manifests should be of the following format: 

# test paths are relative to the manifest
[test1.js]
[test2.js]
[foo/bar/test3.js]
# this test is disabled and won't be run
[test4.js]
disabled = "Doesn't work, see bug 123456"

Alternatively, the --test-path parameter can point to a single test which might be useful for debugging purposes.

Peptest uses ManifestDestiny. See additional documentation.

Test Format

Tests are simply Javascript files that will be executed in chrome space. This means they have access to any of the API's that an extension would normally have access to. In addition to this, they also have access to Mozmill's driver for convenient automation of Firefox/Thunderbird's UI.

It is helpful to think of tests as a series of actions. That is, the UI thread will only be checked for responsiveness while an action is currently happening. This ensures that we are only testing the actions that we care about, and that the test isn't overrun with noise generated during setup or teardown. To accomplish this, tests call a function called 'performAction(name, func)' which takes in two parameters, a name and a function pointer whose contents consist of a single action.

The following example test will make sure that the browser remains responsive while opening a page and while searching in Google. 

// import mozmill and initialize a controller
Components.utils.import('resource://mozmill/driver/mozmill.js');
let controller = getBrowserController();

// Create our first action which will open Google
performAction('open_google', function() {
  controller.open('http://google.com');
  controller.waitForPageLoad();
});

// stuff not inside a performAction() call won't be tested for responsiveness
let textbox = findElement.ID(controller.tabs.activeTab, 'lst-ib');
let button = findElement.Name(controller.tabs.activeTab, 'btnK');

// Create our second action which will perform a search in the google searchbox
performAction('search_google', function() {
  textbox.sendKeys('foobar');
  button.click();
  controller.waitForPageLoad();
});

For documentation on using Mozmill's driver see: The Mozmill Reference Desk

Note that using mozmill may be convenient but is not required. Here is another example. 

let window = getWindow();
let width = window.outerWidth;
let height = window.outerHeight;

performAction('resize_by', function() {
  window.resizeBy(100, 100);
});

performAction('resize_to', function() {
  window.resizeTo(800, 600);
});

// Tests should clean up after themselves
window.resizeTo(width, height);

There are a few example Peptests in the repo.

Log Format

Peptest uses the mozlog module to standardize on the logging format. 

PEP TEST-START | test_sanity.js
PEP TEST-UNEXPECTED-FAIL | test_sanity.js | loadPage | unresponsive time: 116 ms
PEP TEST-UNEXPECTED-FAIL | test_sanity.js | loadPage | unresponsive time: 61 ms
PEP TEST-UNEXPECTED-FAIL | test_sanity.js | loadPage | unresponsive time: 275 ms
PEP TEST-END | test_sanity.js | finished in: 8022 ms
PEP TEST-START | test_sanity2.js
PEP TEST-PASS | test_sanity2.js
PEP TEST-END | test_sanity2.js | finished in: 1536 ms

When a responsiveness measurement runs over 50ms a TEST-UNEXPECTED-FAIL message is output. The second column is the name of the test file, the third column is the name of the particular action that was being performed and the last column is the time in milliseconds that the event took to process.

Further Work

Mozmill e10s

Note: this will likely be put on hold indefinitely until the status of e10s becomes more clear

Mozmill doesn't use MessageManager when it interacts with content. This means that it will not work correctly with Electrolysis builds (which is mostly the entire point of Peptests). Mozmill will need to be refactored to use MessageManager and also to stop using the gBrowser object which doesn't exist in mobile Firefox.

Detailed results

Dietrich has written some tools to determine exactly which Javascript calls were unresponsive. This is a very useful tool which should be incorporated into the Peptest harness. More details can be found here

I (ahal) would really like someone to own bug 580055, un-bitrot the patch and get it checked in (with ifdefs or something). If you want, I'll even un-bitrot the patch for you :)

Buildbot Integration

Work has been done on writing a mozharness peptest script in bug 692091. There is a mozharness script checked in and work is being done to get it running on tryserver.

Related Bugs

API Reference

There is a Peptest API that gets injected into the scope of every test. Each test can use the following methods:

General

performAction(name, function)

Performs an action during which responsiveness measurements are recorded

  • name - the name of the action to run
  • function - a function pointer of the action to run

Logging

For logging messages to terminal

  • msg - the message to log

Logging methods

log.debug(msg)

log.info(msg)

log.warning(msg)

log.error(msg)

Retrieved from "https://wiki.allizom.org/index.php?title=Auto-tools/Projects/peptest&oldid=371658"