TestEngineering/Performance/Talos/Running: Difference between revisions

Jump to navigation Jump to search

TestEngineering/Performance/Talos/Running (view source)

Revision as of 11:46, 16 March 2021

1,840 bytes removed ,  16 March 2021
no edit summary
No edit summary
 
(10 intermediate revisions by 2 users not shown)
Line 1: Line 1:
= Running Talos Tests =
{{DISPLAYTITLE:Running Talos Tests}}
{{todo|add [[../Profiling]]}}


== Locally ==
= Locally =
{{warning|Firefox must have already been built successfully locally. See [https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Simple_Firefox_build Building Firefox] for details.}}


Prerequisite: Firefox has already been built successfully locally.
[https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/mach mach] is used to run the [https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#test-definitions Talos tests] locally. You can run suites of tests or individual tests on their own.
 
[https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/mach Mach] is used to run the [https://wiki.mozilla.org/Buildbot/Talos/Tests Talos tests] locally. You can run suites of tests or individual tests on their own.


For example, to run the 'chromez' test suite:
For example, to run the 'chromez' test suite:


    ./mach talos-test --suite chromez
  ./mach talos-test --suite chromez


To run a single test (the 'tart' test):
To run a single test (the 'tart' test):


    ./mach talos-test --activeTests tart
  ./mach talos-test --activeTests tart


Certain talos suites/tests require external pagesets to be downloaded. If pagesets are required, they will be downloaded and extracted automatically when running talos via mach. To disable this automatic pageset downloading, add the 'no-download' argument:
'''Known Issues'''


    ./mach talos-test --suite g1 --no-download
In some cases, the Talos setup may not finish properly, throwing ''Import Error: No module named'' for packages such as ''wptserve'' and ''simplejson''. This is known to affect MacOS users: [https://bugzilla.mozilla.org/show_bug.cgi?id=1667189 Bug 1667189 - Talos test refuses to run because of missing python packages]


To see a list of available talos suites:
If you encounter this error, running the following command should resolve it:


    ./mach talos-test --print-suites
  brew link python3


To see a list of available talos tests:
Otherwise, you can manually activate Talos's virtual environment (found in your ''obj'' directory, which differs based on your machine architecture and OS, i.e. ''obj-x86_64-apple-darwin19.6.0'') and install the missing packages:


    ./mach talos-test --print-tests
  source mozilla-central/obj-[your directory suffix]/testing/talos-venv/bin/activate
  pip install wptserve simplejson


Further help running talos via mach:
Certain Talos suites/tests require external pagesets to be downloaded. If pagesets are required, they will be downloaded and extracted automatically when running Talos via mach. To disable this automatic pageset downloading, add the 'no-download' argument:


    ./mach talos-test --help
  ./mach talos-test --suite g1 --no-download


'''Tip:''' Clear the operating system cache between runs, in order to significantly improve the consistency of the results. For example, on linux:
To see a list of available Talos suites:


    find . | xargs touch
  ./mach talos-test --print-suites


== Try Server ==
To see a list of available Talos tests:


To run talos tests on the try server, you just simply push to try as normal, specifying that you want to run Talos tests. It is highly recommended that you use [http://trychooser.pub.build.mozilla.org/ trychooser] to build your try server syntax (helpful hint for using trychooser: in the list of talos suites available you can hover over any one of the suites, and a list of tests included in that suite will be displayed).
  ./mach talos-test --print-tests


To run all of the talos tests as part of your try push, just add "-t all" to the end of your try syntax. To run a single talos suite as part of your try push, just add "-t <name of suite>" to your try syntax (i.e. "-t g1").
Further help running talos via mach:


To turn on [https://wiki.mozilla.org/Buildbot/Talos/Profiling profiling] as part of your talos tests on try, add "mozharness: --spsProfile" to the end of the try syntax (i.e. "-t g1 mozharness: --spsProfile"). There is an option to do this in trychooser.
  ./mach talos-test --help


When viewing your talos try server results on Treeherder, look for the group names 'T' and 'T-e10s'. Click on the green symbol corresponding to the talos suite name, and then click on the 'Performance' tab below to see the test results.
'''Tip:''' Clear the operating system cache between runs, in order to significantly improve the consistency of the results. For example, on linux:
 
If you're curious about the hardware we use, here is a [https://wiki.mozilla.org/Buildbot/Talos/Misc#Hardware_Profile_of_machines_used_in_automation description of the specs].  You can also refer to the [https://wiki.mozilla.org/ReferencePlatforms#Build_Reference_Platforms test reference platforms] documentation.
 
== Are my Talos Results OK? ==
 
The best way to answer this question is to push to try server and compare the performance of your push against a baseline, using Perfherder (it is recommended hitting retrigger a few times to get at least 5 data points for each test). After pushing to try with talos tests enabled, you will receive an email with a link to Perfherder where you can compare your results. Make sure you look at the different platforms.
 
= Hacking on Talos Itself =
 
== Dev Environment ==
 
Prerequisites:
 
Local clone of mozilla-central and a successful [https://developer.mozilla.org/en-US/docs/Simple_Firefox_build mozilla-central Firefox build] completed.
 
On Linux, setuptools installed:
 
    apt-get install python-setuptools
 
Not required on Windows. For other operating systems see [https://pypi.python.org/pypi/setuptools Setuptools].
   
On Linux, virtualenv installed:
 
    pip install virtualenv
 
Not required on Windows. For other operating systems see [http://www.virtualenv.org/en/latest/virtualenv.html#installation Virtualenv].
 
Setup talos (linux):
 
    cd mozilla-central/testing/talos
    virtualenv .
    . bin/activate
    python setup.py develop
 
Setup talos (Windows with mozilla-build):
 
    cd mozilla-central/testing/talos
    python INSTALL.py  # only required once or after updating mozilla-build
    source Scripts/activate  # you can also use '.' (dot) instead of 'source'
 
== Testing your Talos Patch Locally ==
 
Testing locally involves running some subset of the Talos tests on
desktop and possibly mobile.  Obviously not all permutations of tests
and the ways of running them can be tried, so common sense should be
used as to what is run.  You may also want to run Talos' internal
unittests: http://hg.mozilla.org/build/talos/file/tip/tests
 
You should tailor your choice of tests to pick those that cover what
you've changed programmatically, but in general you should probably
run at least one startup test and one pageloader test.  A good
baseline might be the 'tresize' and 'tsvgx' test suites.
 
Run the tests from mozilla-central/testing/talos (with the venv activated, as noted above):


talos --develop --executablePath pathtofirefox --activeTests tart
  find . | xargs touch


The '--develop' option indicates to run in develop mode and store the results in local.json + local.out.
= Try server =
{{note|It's recommended to run ~5 times using the <code>--rebuild</code> option due to the noise inherent in performance test results}}


The '--executablePath' option tells Talos where the firefox installation we want to run is located (i.e. '~/mozilla/objdir/dist/bin/firefox' or whatever the full path is to your firefox executable that will be running the tests).
Talos tests can be triggered using the [https://firefox-source-docs.mozilla.org/tools/try/ try server]. The following command will run all Talos tests against all available platforms five times using the [https://firefox-source-docs.mozilla.org/tools/try/selectors/fuzzy.html fuzzy selector]:


The '--activeTests' argument provides is a list of tests we want to run (if running multiple tests, separate each test name with ':').
  ./mach try fuzzy --query "^test- 'talos" --rebuild 5


For a list of complete options:
{{todo|update this}}
To turn on [[../Profiling|profiling]] as part of your talos tests on try, add "mozharness: --spsProfile" to the end of the try syntax (i.e. "-t g1 mozharness: --spsProfile"). There is an option to do this in trychooser.


    talos --help
When viewing your Talos try server results on Treeherder, look for the group names 'T' and 'T-e10s'. Click on the green symbol corresponding to the Talos suite name, and then click on the 'Performance' tab below to see the test results.


== Testing your Talos Patch on Try Server ==
If you're curious about the hardware we use, here is a [[../Platforms|description of the specs]].  You can also refer to the [[ReferencePlatforms#Build_Reference_Platforms|test reference platforms]] documentation.


Just push your talos patch to the try server. See [https://wiki.mozilla.org/Buildbot/Talos/Running#Try_Server the Try Server section] above.
= Are my Talos results OK? =
* [duplicated & updated from old page duplicated & updated from old page]
The best way to answer this question is to push to try server and compare the performance of your push against a baseline, using Perfherder (it is recommended hitting retrigger a few times to get at least 5 data points for each test). After pushing to try with Talos tests enabled, you will receive an email with a link to Perfherder where you can compare your results. Make sure you look at the different platforms.
Davehunt
Confirmed users
2,177

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/1215917...1234444"

Navigation menu