TestEngineering/Performance/RunningTests

Guide to running performance tests

Prerequisites

What?

To be able to run one or more performance tests you need to know which are those. The Performance Test Projects from Performance Test Engineering page contains the projects and their test list.

How?

There’s a tool called mach we use to run the tests, but you need to at least have phabricator and bugzilla accounts to be able to run on try.

Running tests

Where?

As long as you have mach installed and configured, you can run a performance test locally or on try. But, first command you need to run to get familiar with the complexity of mach is:
$ ./mach help

On try

here’s the option on running a performance test on try (treeherder). This option will run the test on an integration environment from mozilla, meaning that it won’t use any of your workstation resources to get the results.

The command is:
$ ./mach try chooser
to get a nice UI to choose the tests to run, platform, build type

or
$ ./mach try fuzzy
to get a CLI on your console to choose the tests you want to run.


If you know exactly how to spell some words in the test signature, like platform, test name or build type, you might want to consider using the simple quote ‘ to filter up the results to your preference.


If you know exactly how the signature starts you can use a ^ in front of your search string and the results will narrow down to just that signature


If you want to skip the chooser or the fuzzy and to push the test to try directly from the command line, you can use the option -q (from query). The option --no-push will allow you to check the jobs you’re pushing by printing them to the command line without actually pushing them to try. The option -m is pretty handy when you push several times the same test (and believe me, you will!).
$ ./mach try fuzzy --full -q="'linux64 'raptor-tp6-1-firefox" -m="base for [bug/commit]" --no-push


And narrowing the results down
$ ./mach try fuzzy --full -q="'linux64-shippable/opt-raptor-tp6-1-firefox" -m="base for [bug/commit]" --no-push


Or just specifying how the test signature starts:


Once you found your query, you just need to remove the --no-push to push to try.

Those are just some examples of what performance sheriffs do when pushing tests to try. You can do anytime the command below to get the full help of the try command.
$ ./mach try --help