Buildbot/Talos/Running: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(42 intermediate revisions by 13 users not shown)
Line 1: Line 1:
#REDIRECT [[TestEngineering/Performance/Talos/Running]]
= Running Talos Tests =
= Running Talos Tests =


== I have a patch for Talos, what tests to I run? ==
== Locally ==
 
Prerequisite: Firefox has already been built successfully locally.
 
[https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/mach Mach] is used to run the [https://wiki.mozilla.org/Buildbot/Talos/Tests Talos tests] locally. You can run suites of tests or individual tests on their own.


If you are making changes to talos obviously running the tests locally will be the first step.  The next logical step is to run the tests on Try server (try: -b o -p all -u none -t all).
For example, to run the 'chromez' test suite:


=== Testing Locally ===
    ./mach talos-test --suite chromez


Testing locally involves running some subset of the Talos tests on
To run a single test (the 'tart' test):
desktop and possibly mobile. Obviously not all permutations of tests
 
and the ways of running them can be tried, so common sense should be
    ./mach talos-test --activeTests tart
used as to what is run.  You may also want to run Talos' internal
 
unittests: http://hg.mozilla.org/build/talos/file/tip/tests
To run two single tests, use the ':' to separate the test list:
 
    ./mach talos-test --activeTests tart:damp
 
Certain talos suites/tests require external pagesets to be downloaded. If pagesets are required, they will be downloaded and extracted automatically when running talos via mach. To disable this automatic pageset downloading, add the 'no-download' argument:
 
    ./mach talos-test --suite g1 --no-download
 
To see a list of available talos suites:


You should tailor your choice of tests to pick those that cover what
    ./mach talos-test --print-suites
you've changed programmatically, but in general you should probably
run at least one startup test and one pageloader test.  A good
baseline might be:


  # refer to [https://wiki.mozilla.org/Buildbot/Talos#Running_locally_-_Source_Code running locally] for more details
To see a list of available talos tests:
  talos -n -d -a ts:tsvg -e `which firefox` --develop --datazilla-url output.json --mozAfterPaint


=== Testing on Try Server ===
    ./mach talos-test --print-tests


To speed up your development time and everybody else who uses Try server, there is no need to run all tests on all platforms unless this is a major change.  Here are some guidelines to follow for testing patches to talos (or dependent modules):
Further help running talos via mach:


* If you patch touches one file and is very minor, get minimal testing on it (1 mobile test, 1 windows test, local testing): try: -b o -p win32,android -u none -t svgr,remote-tsvg
    ./mach talos-test --help
* If you patch affects setup of the tests (config or profile) or launching of the process and is very minimal, get minimal testing on it (1 mobile test, 1 windows test, local testing): try: -b o -p win32,android -u none -t svgr,remote-tsvg
* If your patch changes a test (add or edit), test that test locally (remember a new test will need graph server changes and buildbot changes)
* If your patch changes mobile testing, test all mobile tests and 1 desktop test: (There is no good current way of doing this with try;  I recommend two try runs:


try: -b o -p win32 -u none -t svgr
'''Tip:''' Clear the operating system cache between runs, in order to significantly improve the consistency of the results. For example, on linux:


try: -b o -p android -u none -t all
    find . | xargs touch


)
== Try Server ==
* If your patch changes results or output processing, run ts, tp5 on mobile, windows and locally: try: -b o -p win32,android -u none -t tpn,chromez,remote-ts,remote-tp4m_nochrome
* If your patch changes any import statement (or code referenced in the import), please test on windows, linux, and mobile as these all run different versions of python: try: -b o -p linux,win32,android -u none -t all
* If your patch changes a lot of stuff around, if you are not sure, or if you are going to deploy a new talos.zip, it is strongly recommended to run all talos tests: try: -b o -p all -u none -t all


If you are reviewing a talos patch, it is your responsibility to recommend the proper testing approach.  If you think it needs more than these guidelines, call it out.  If it needs less, call it out also.
To run talos tests on the try server, you just simply push to try as normal, specifying that you want to run Talos tests. It is highly recommended that you use [http://trychooser.pub.build.mozilla.org/ trychooser] to build your try server syntax (helpful hint for using trychooser: in the list of talos suites available you can hover over any one of the suites, and a list of tests included in that suite will be displayed).


== Are my numbers ok ==
To run all of the talos tests as part of your try push, just add "-t all" to the end of your try syntax. To run a single talos suite as part of your try push, just add "-t <name of suite>" to your try syntax (i.e. "-t g1").


The best way to answer this question is to push to try server and compare the reported numbers from the logs (use tbpl as a log parser) and compare that with the [[http://graphs.mozilla.org/ graph server]].  I recommend using tbpl to open the link to the graphs.
To turn on [https://wiki.mozilla.org/Buildbot/Talos/Profiling profiling] as part of your talos tests on try, add "mozharness: --spsProfile" to the end of the try syntax (i.e. "-t g1 mozharness: --spsProfile"). There is an option to do this in trychooser.


If you are planning on landing on mozilla-central, look at tests from mozilla-central. Be aware of PGO vs Non PGO and Chrome vs Non Chrome.  TBPL makes this a moderately pain free process (i.e. about 30 minutes).  This is one of the big problems we are solving with datazilla.
When viewing your talos try server results on Treeherder, look for the group names 'T' and 'T-e10s'. Click on the green symbol corresponding to the talos suite name, and then click on the 'Performance' tab below to see the test results.


== Using try server ==
If you're curious about the hardware we use, here is a [https://wiki.mozilla.org/Buildbot/Talos/Misc#Hardware_Profile_of_machines_used_in_automation description of the specs].  You can also refer to the [https://wiki.mozilla.org/ReferencePlatforms#Build_Reference_Platforms test reference platforms] documentation.


If you have access to generate try builds you can also have performance tests run against a custom version of talos. The performance results will be generated on the same machines that generate the talos results for all check-ins on all branches. This involves a few steps:
== Are my Talos Results OK? ==


* Run create_talos_zip.py from the root of your talos directory and upload the file somewhere that the build system can find it (e.g. http://people.mozilla.org/~wlachance/talos.zip)
The best way to answer this question is to push to try server and compare the performance of your push against a baseline, using Perfherder (it is recommended hitting retrigger a few times to get at least 5 data points for each test). After pushing to try with talos tests enabled, you will receive an email with a link to Perfherder where you can compare your results. Be aware of the options: e.g. pgo vs non-pgo or e10s vs non-e10s.
* Check out a copy of Mozilla central
* Modify the file "testing/talos/talos.json" to point to the copy of talos you uploaded earlier
* [https://wiki.mozilla.org/ReleaseEngineering/TryServer#How_to_push_to_try Push] this change to try server using the right syntax (you can use TryChooser to help with this: recommended is to test talos thoroughly, but standard unit tests can be skipped)


A bit more information can be found in this blog post from armenzg: http://armenzg.blogspot.com/2011/12/taloszip-talosjson-and-you.html
= Hacking on Talos Itself =


== Running locally - Source Code ==
== Dev Environment ==


'''http://hg.mozilla.org/build/talos/archive/tip.tar.gz may be installed in the usual manner for a python package (easy_install, etc).  However, `pip` on windows fails due to the pywin32 dependency.  See {{bug|787496}}'''
Prerequisites:


For the majority of the tests, we include test files and tools out of the box. We need to do these things:
Local clone of mozilla-central and a successful [https://developer.mozilla.org/en-US/docs/Simple_Firefox_build mozilla-central Firefox build] completed.


* '''clone talos:'''
On Linux, setuptools installed:
hg clone http://hg.mozilla.org/build/talos


* '''run the [http://hg.mozilla.org/build/talos/file/tip/INSTALL.py install script]''' which
    apt-get install python-setuptools
cd talos
python INSTALL.py
# (Ignore errors like "fatal error: 'yaml.h' file not found. To get rid of this error message you can download and install PyYAML from http://pyyaml.org/wiki/PyYAML but it is not necessary.


** creates a [https://developer.mozilla.org/en/Python/Virtualenv Virtualenv] in the same directory as 'INSTALL.py'
Not required on Windows. For other operating systems see [https://pypi.python.org/pypi/setuptools Setuptools].
** installs the talos [https://developer.mozilla.org/en/Python#Python_packaging python package] into the virtualenv, including its [https://wiki.mozilla.org/Auto-tools/Projects/MozBase MozBase dependencies] via 'python setup.py develop'
   
** ''(alternatively, you can perform these steps yourself)''
On Linux, virtualenv installed:
* '''activate the virtualenv:'''
''(on windows):''
Scripts\activate.bat


''(on osx/linux)'':
    pip install virtualenv
. bin/activate


* unpack a copy of firefox somewhere (for this example, we'll use `which firefox` as if firefox was on your path)
Not required on Windows. For other operating systems see [http://www.virtualenv.org/en/latest/virtualenv.html#installation Virtualenv].
* '''setup a webserver''' if you're not using the '--develop' flag (WE STRONGLY RECOMMEND USING THE --develop FLAG)
** setup apache or similar webserver to have http://localhost -> the talos subdirectory of the talos checkout
** alternatively you can use the --develop flag to PerfConfigurator which configures to use a python webserver, [https://github.com/mozilla/mozbase/tree/master/mozhttpd mozhttpd], as shown below


* '''run tests:'''
Setup talos (linux):
talos -n -d --develop --executablePath pathtofirefox --activeTests ts --results_url ts.txt --datazilla-url ts.json --output ts_desktop.yml --mozAfterPaint
** --develop indicates to run in develop mode and to set up a webserver for you
** --executablePath tells Talos where the firefox installation we want to run is located
*** we have pathtofirefox as an example, you can use '~/mozilla/objdir/dist/bin/firefox' or whatever the full path is to your firefox executable that will be running the tests.
** --activeTests is a list of tests we want to run separated by ':'.  In this example, we are running the startup test, ts.
** --results_url indicates a HTTP URL to POST results to or a file to append to
** --output is the new config file we want to generate
You can use `talos --help` to get a complete list of options


If you're looking to run remote talos, instructions are at: https://wiki.mozilla.org/Mobile/Fennec/Android#talos
    cd mozilla-central/testing/talos
    virtualenv .
    . bin/activate
    python setup.py develop


We do not include the tp5 or similar pagesets for legal restrictions.
Setup talos (Windows with mozilla-build):


Talos will refuse to run if you have an open browser.
    cd mozilla-central/testing/talos
    python INSTALL.py  # only required once or after updating mozilla-build
    source Scripts/activate  # you can also use '.' (dot) instead of 'source'


If you want to load an extension while running Talos, you want a few more command line arguments:
== Testing your Talos Patch Locally ==


  talos -n -d --executablePath=../firefox/firefox --sampleConfig=sample.config --activeTests=ts --extension=mozmill-1.5.2-sb+tb+fx+sm.xpi --addonID=mozmill@mozilla.com --output=my.config
Testing locally involves running some subset of the Talos tests on
desktop and possibly mobile. Obviously not all permutations of tests
and the ways of running them can be tried, so common sense should be
used as to what is run. You may also want to run Talos' internal
unittests: http://hg.mozilla.org/build/talos/file/tip/tests


* --extension is the file of the XPI we want to install in the profile
You should tailor your choice of tests to pick those that cover what
* --addonID is the ID of the addon to install
you've changed programmatically, but in general you should probably
run at least one startup test and one pageloader test.  A good
baseline might be the 'tresize' and 'tsvgx' test suites.


== How Talos is Run in Production ==
Run the tests from mozilla-central/testing/talos (with the venv activated, as noted above):


* buildbot constructs commands to launch [http://hg.mozilla.org/build/talos/file/tip/talos/PerfConfigurator.py PerfConfigurator] from a config file: http://hg.mozilla.org/build/buildbot-configs/raw-file/tip/mozilla-tests/config.py ; there are a number of suites, each of which may contain multiple tests
talos --develop --executablePath pathtofirefox --activeTests tart


* a slave invokes [http://hg.mozilla.org/build/talos/file/tip/talos/run_tests.py run_tests.py] on the generated [http://www.yaml.org/ YAML] Talos configuration
The '--develop' option indicates to run in develop mode and store the results in local.json + local.out.


* Talos will run the tests and measured results which are uploaded to [http://graphs.mozilla.org/ graphserver] (after being suitably averaged per-page for Pageloader tests)
The '--executablePath' option tells Talos where the firefox installation we want to run is located (i.e. '~/mozilla/objdir/dist/bin/firefox' or whatever the full path is to your firefox executable that will be running the tests).
** (Talos also uploads raw results to [https://github.com/mozilla/datazilla datazilla])


* the graphserver performs averaging across the pageset (for Pageloader tests) or across cycles (for Startup tests) and returns a number via HTTP to Talos which is then printed to the log
The '--activeTests' argument provides is a list of tests we want to run (if running multiple tests, separate each test name with ':').


* [https://tbpl.mozilla.org/ TBPL] receives the name of the suite from buildbot.  These are correlated to TBPL letters via http://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/file/tip/js/Config.js . The computed results from graphserver are scraped from the log and displayed when a TBPL suite letter is clicked on in the lower right corner
For a list of complete options:


=== Running Tp the Automation Way ===
    talos --help
In our automation, we run beneath many more restrictions than normal users.  One of those restrictions is that our automation machines are walled off from the real-world networks.  Because of this, and because we want to test pure page-loading and rendering time of Firefox, we serve the pages from localhost using [http://httpd.apache.org/ Apache] thus eliminating all network latency and uncertainty.  You've probably noticed this if you looked at the talos/page_load_test/tp4.manifest.


To do this, we construct full downloads of sites in our manifest and they are placed on the automation slave at run time.  Because we cannot at this time distribute our full page load test set, I'll walk through how these are set up and show you how to make your own.  Note that our next version of the page load set will be distributable, so soon this won't be an issue.
== Testing your Talos Patch on Try Server ==


In the meantime, here's the instructions:
Just push your talos patch to the try server. See [https://wiki.mozilla.org/Buildbot/Talos/Running#Try_Server the Try Server section] above.
<ol>
<li>Use [http://people.mozilla.org/~ctalbert/talos/makepagebundle.sh this script] or, you use the following wget command to fetch a page and everything it links to in order to have a complete page for offline use: <pre> $> wget -p -k -H -E -erobots=off --no-check-certificate -U "Mozilla/5.0 (firefox)" --restrict-file-names=windows --restrict-file-names=nocontrol $URL -o outputlog.txt</pre>
</li>
<li>Once you have a cache of pages, install Apache: <pre> $> sudo apt-get install apache2</pre></li>
<li>Copy your page into the proper location for Apache to serve it. Note that I like to create a page_load_test directory to separate talos from anything else on the webserver.  So with Apache defaults, that's something like: <pre> $> mkdir /var/www/page_load_test; cp -R <dir> /var/www/page_load_test/.</pre></li>
<li>Now, add the local URL into your manifest: <pre>http://localhost/page_load_test/<dir></pre></li>
<li>Run the tp tests as above, pointing the config file at your manifest.
</li>
</ol>

Latest revision as of 13:02, 6 August 2019

Redirect to:

Running Talos Tests

Locally

Prerequisite: Firefox has already been built successfully locally.

Mach is used to run the Talos tests locally. You can run suites of tests or individual tests on their own.

For example, to run the 'chromez' test suite: 

   ./mach talos-test --suite chromez

To run a single test (the 'tart' test): 

   ./mach talos-test --activeTests tart

To run two single tests, use the ':' to separate the test list: 

   ./mach talos-test --activeTests tart:damp

Certain talos suites/tests require external pagesets to be downloaded. If pagesets are required, they will be downloaded and extracted automatically when running talos via mach. To disable this automatic pageset downloading, add the 'no-download' argument: 

   ./mach talos-test --suite g1 --no-download

To see a list of available talos suites: 

   ./mach talos-test --print-suites

To see a list of available talos tests: 

   ./mach talos-test --print-tests

Further help running talos via mach: 

   ./mach talos-test --help

Tip: Clear the operating system cache between runs, in order to significantly improve the consistency of the results. For example, on linux: 

    find . | xargs touch

Try Server

To run talos tests on the try server, you just simply push to try as normal, specifying that you want to run Talos tests. It is highly recommended that you use trychooser to build your try server syntax (helpful hint for using trychooser: in the list of talos suites available you can hover over any one of the suites, and a list of tests included in that suite will be displayed).

To run all of the talos tests as part of your try push, just add "-t all" to the end of your try syntax. To run a single talos suite as part of your try push, just add "-t <name of suite>" to your try syntax (i.e. "-t g1").

To turn on profiling as part of your talos tests on try, add "mozharness: --spsProfile" to the end of the try syntax (i.e. "-t g1 mozharness: --spsProfile"). There is an option to do this in trychooser.

When viewing your talos try server results on Treeherder, look for the group names 'T' and 'T-e10s'. Click on the green symbol corresponding to the talos suite name, and then click on the 'Performance' tab below to see the test results.

If you're curious about the hardware we use, here is a description of the specs. You can also refer to the test reference platforms documentation.

Are my Talos Results OK?

The best way to answer this question is to push to try server and compare the performance of your push against a baseline, using Perfherder (it is recommended hitting retrigger a few times to get at least 5 data points for each test). After pushing to try with talos tests enabled, you will receive an email with a link to Perfherder where you can compare your results. Be aware of the options: e.g. pgo vs non-pgo or e10s vs non-e10s.

Hacking on Talos Itself

Dev Environment

Prerequisites:

Local clone of mozilla-central and a successful mozilla-central Firefox build completed.

On Linux, setuptools installed: 

    apt-get install python-setuptools

Not required on Windows. For other operating systems see Setuptools.

On Linux, virtualenv installed: 

    pip install virtualenv

Not required on Windows. For other operating systems see Virtualenv.

Setup talos (linux): 

    cd mozilla-central/testing/talos
    virtualenv .
    . bin/activate
    python setup.py develop

Setup talos (Windows with mozilla-build): 

    cd mozilla-central/testing/talos
    python INSTALL.py  # only required once or after updating mozilla-build
    source Scripts/activate  # you can also use '.' (dot) instead of 'source'

Testing your Talos Patch Locally

Testing locally involves running some subset of the Talos tests on desktop and possibly mobile. Obviously not all permutations of tests and the ways of running them can be tried, so common sense should be used as to what is run. You may also want to run Talos' internal unittests: http://hg.mozilla.org/build/talos/file/tip/tests

You should tailor your choice of tests to pick those that cover what you've changed programmatically, but in general you should probably run at least one startup test and one pageloader test. A good baseline might be the 'tresize' and 'tsvgx' test suites.

Run the tests from mozilla-central/testing/talos (with the venv activated, as noted above): 

talos --develop --executablePath pathtofirefox --activeTests tart

The '--develop' option indicates to run in develop mode and store the results in local.json + local.out.

The '--executablePath' option tells Talos where the firefox installation we want to run is located (i.e. '~/mozilla/objdir/dist/bin/firefox' or whatever the full path is to your firefox executable that will be running the tests).

The '--activeTests' argument provides is a list of tests we want to run (if running multiple tests, separate each test name with ':').

For a list of complete options: 

    talos --help

Testing your Talos Patch on Try Server

Just push your talos patch to the try server. See the Try Server section above.

Retrieved from "https://wiki.allizom.org/index.php?title=Buildbot/Talos/Running&oldid=1216058"