TestEngineering/Performance/RunningTests

Prerequisites

To run the tests you will need to have the source code available locally, and satisfy the requirements for running mach commands. If you want to run tests against the same infrastructure as our continuous integration then you will need to follow the try server documentation.

Identifying tests to run

To be able to run one or more performance tests you first need to know which tests you would like to run. If you're responding to a regression bug then the test names will be listed in the bug report. The performance tests and frameworks available can be found from our projects list. You can also see our new performance test documentation, which will eventually replace our wiki pages.

Running tests locally

[TODO]

Scheduling tests on try

Follow this guide to schedule jobs against the try server.

Rebuilds

Due to the variance in performance test results it is a good idea to schedule multiple rebuilds. We typically recommend 3 rebuilds. This can be achieved by adding --rebuild 3 to your try syntax.

Presets

If you're unsure which tests to run, there are some mach try presets that can help:

perf

Runs all performance (raptor and talos) tasks across all platforms. Android hardware platforms are excluded due to resource limitations.

perf-chrome

Runs the talos tests most likely to change when making a change to the browser chrome. This skips a number of talos jobs that are unlikely to be affected in order to conserve resources.

Scheduling hidden jobs

Some jobs are hidden by default to reduce them being scheduled unintentionally. These are typically jobs that run on limited pools of hardware such as mobile devices. To make these available to your try run add the --full option.

Viewing test results

[TODO] Treeherder, Perfherder graphs

Comparing results from multiple try jobs

[TODO] Perfherder compare view