Buildbot/Talos/Running

Running Talos Tests

Using mach

First, make sure that firefox is already compiled. Then, execute the mach command in the top source directory. Let’s try running the chromez test suite. The list of valid suites are in the talos.json file.

   ./mach talos-test --suite chromez

I have a patch for Talos, what tests to I run?

If you are making changes to talos obviously running the tests locally will be the first step. The next logical step is to run the tests on Try server (try: -b o -p all -u none -t all).

Testing Locally

Testing locally involves running some subset of the Talos tests on desktop and possibly mobile. Obviously not all permutations of tests and the ways of running them can be tried, so common sense should be used as to what is run. You may also want to run Talos' internal unittests: http://hg.mozilla.org/build/talos/file/tip/tests

You should tailor your choice of tests to pick those that cover what you've changed programmatically, but in general you should probably run at least one startup test and one pageloader test. A good baseline might be:

 # refer to running locally for more details
 talos -n -d -a tresize:tsvgx -e `which firefox` --develop

Testing on Try Server

There are three scenarios that are needed for testing talos on try server:

• Firefox Developer: testing your Firefox changes for performance issues
• Talos Validation: testing your already reviewed and landed patches to the Talos toolchain in production
• Talos Developer: testing patches to Talos which have not been reviewed or landed

For the purposes of Talos development we will ignore the Firefox Developer scenario and discuss the two scenarios regarding Talos toolchain development. In fact, the process is similar for both, lets get started:

First off, you need to edit [testing/talos/talos.json] in your Firefox tree, what is of most importance is:

   "talos.zip": {
       "url": "http://talos-bundles.pvt.build.mozilla.org/zips/talos.49b74c08dad4.zip",
       "path": ""
   },
   "global": {
       "talos_repo": "https://hg.mozilla.org/build/talos",
       "talos_revision": "5e18c3e56875"
   }

Android 4.0 tests use "talos.zip:, and all desktop platforms use the "global" section. You can see in global there are 2 fields, "talos_repo" and "talos_revision".

In the Talos Validation scenario, you only need to change "talos_revision" to be the revision of [talos] that your change is.

For the Talos Developer scenario, you would edit "global" to look like this:

   "global": {
       "talos_repo": "http://hg.mozilla.org/users/jmaher_mozilla.com/tpain",
       "talos_revision": "default"
   }

There are 2 key points here:

* the repo is pointing to a [user repo] which contains a full copy of the talos repo and all local changes made
* the revision is pointing at "default"

NOTE: there is a trick with using "default" as the revision, it allows you to check in changes and then retrigger a job on try. This avoids the need to push a new revision to try and wait for a build.

Finally if you are validating a push or adjusting something that should or will affect Android Talos, then you need to [create a new talos.zip], ftp that to a people.mozilla.org directory, and then edit the talos.zip section of talos.json to reference it.

Once you have talos.json changed, you just need to select the proper [try chooser] syntax.

Are my numbers ok

The best way to answer this question is to push to try server and compare the reported numbers from the logs (use Treeherder as a log parser) and compare that with the [graph server]. I recommend using Treeherder to open the link to the graphs.

If you are planning on landing on mozilla-central, look at tests from mozilla-central. Be aware of PGO vs Non PGO and e10s vs Non e10s. Treeherder makes this a moderately pain free process (i.e. about 30 minutes).

Using try server

If you have access to generate try builds you can also have performance tests run against a custom version of talos. The performance results will be generated on the same machines that generate the talos results for all check-ins on all branches. This involves a few steps:

Testing changes to talos properly

• Create a user repo with talos, commit and push your changes
• Check out a copy of Mozilla central
• Modify the file "testing/talos/talos.json" to point to your user repo and put the changeset as 'default'.
  • For android, change the location of talos.zip

Pushing to try

• Push this change to try server using the right syntax (you can use TryChooser to help with this: recommended is to test talos thoroughly, but standard unit tests can be skipped)
• Try server will give you results for a single data point (I recommend hitting retrigger a few times to get at least 3 data points for each test). Graph server is not very useful for viewing your try push data, we have two options for aiding in this effort:
  • [compare.py]: clone talos, setup the virtualenv, run "python compare.py --help", normally I run "python compare.py --revision <rev>" where <rev> is the revision of my try push
  • [compare-talos]: Browse to the url, enter your try server revision, then enter a revision of the branch you are comparing to, you can then browser the delta for each summary and individual test page.

A bit more information can be found in this blog post from armenzg.

Running locally - Source Code

• Note: you must first have setuptools installed on your system

* apt-get install python-setuptools # linux
* Windows with mozilla-build - already included.
* https://pypi.python.org/pypi/setuptools # download and install for all operating systems

• you also need virtualenv installed on your system

* pip install virtualenv # http://www.virtualenv.org/en/latest/virtualenv.html#installation
* Windows with mozilla-build - not required.

For the majority of the tests, we include test files and tools out of the box. You will need the mozilla-central code base:

• clone talos:

hg clone http://hg.mozilla.org/mozilla-central

• setup talos (non-Windows)

cd mozilla-central/testing/talos
virtualenv .
. bin/activate (On Windows: Scripts\activate.bat or . Scripts/activate)
python setup.py develop

• setup talos (Windows with mozilla-build)

cd mozilla-central/testing/talos
python INSTALL.py  # only required once or after updating mozilla-build
source Scripts/activate  # you can also use '.' (dot) instead of 'source'

• run tests:

talos --develop --executablePath pathtofirefox --activeTests tart
# Note: this will open and close the browser real fast for the warmup, then run the test

• • --develop indicates to run in develop mode and store the results in local.json + local.out
  • --executablePath tells Talos where the firefox installation we want to run is located
    • we have pathtofirefox as an example, you can use '~/mozilla/objdir/dist/bin/firefox' or whatever the full path is to your firefox executable that will be running the tests.
  • --activeTests is a list of tests we want to run separated by ':'. In this example, we are running the tart test .
  • --output is the new config file we want to generate

You can use `talos --help` to get a complete list of options

Tip: Run find . | xargs touch to clear operating system caches between talos runs. This significantly improves the consistency of the results.

If you're looking to run remote talos, instructions are at: https://wiki.mozilla.org/Mobile/Fennec/Android#talos

talos --print-tests may be used to display the descriptions and run-time parameters of tests

We do not include the tp5 or similar pagesets for legal restrictions.

How Talos is Run in Production

• buildbot constructs commands to launch mozharness, ex:

/tools/buildbot/bin/python scripts/scripts/talos_script.py --suite chromez --add-option --webServer,localhost --branch-name Mozilla-Inbound-Non-PGO --system-bits 64 --cfg talos/linux_config.py --download-symbols ondemand --use-talos-json --blob-upload-branch Mozilla-Inbound-Non-PGO

• mozharness talos.py downloads common tools, the talos source, pagesets and run the install commands.

• talos run_tests.py is invoked with commandline options which then runs the test as you would locally

• Talos will run the tests and measured results which are uploaded to graphserver (after being suitably averaged per-page for Pageloader tests)
  • (Talos also uploads raw results to PerfHerder)

• the graphserver performs averaging across the pageset (for Pageloader tests) or across cycles (for Startup tests) and returns a number via HTTP to Talos which is then printed to the log

• Treeherder receives the name of the suite from buildbot. These are correlated to Treeherder letters via [1]. The computed results from graphserver are scraped from the log and displayed when a Treeherder suite letter is clicked.

The Test Machines

Here is a [description of the specs] for the machines we use. For more information, please see the Test Reference Platforms.

Running Tp the Automation Way

In our automation, we run beneath many more restrictions than normal users. One of those restrictions is that our automation machines are walled off from the real-world networks. Because of this, and because we want to test pure page-loading and rendering time of Firefox, we serve the pages from localhost using a built in python webserver thus eliminating all network latency and uncertainty. You've probably noticed this if you looked at the talos/page_load_test/tp4.manifest.

To do this, we construct full downloads of sites in our manifest and they are placed on the automation slave at run time. Because we cannot at this time distribute our full page load test set, I'll walk through how these are set up and show you how to make your own. Note that our next version of the page load set will be distributable, so soon this won't be an issue.

In the meantime, here's the instructions:

1. Use the following wget command to fetch a page and everything it links to in order to have a complete page for offline use:

   wget -p -k -H -E -erobots=off --no-check-certificate -U "Mozilla/5.0 (firefox)" --restrict-file-names=windows --restrict-file-names=nocontrol $URL -o outputlog.txt