|
|
| (19 intermediate revisions by 8 users not shown) |
| Line 1: |
Line 1: |
| | #REDIRECT [[TestEngineering/Performance/Talos/Running]] |
| | |
| = Running Talos Tests = | | = Running Talos Tests = |
|
| |
|
| == Using mach == | | == Locally == |
|
| |
|
| First, make sure that firefox is already compiled. Then, execute the mach command in the top source directory. Let’s try running the chromez test suite. The list of valid suites are in the [http://mxr.mozilla.org/mozilla-central/source/testing/talos/talos.json talos.json] file.
| | Prerequisite: Firefox has already been built successfully locally. |
|
| |
|
| ./mach talos-test chromez
| | [https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/mach Mach] is used to run the [https://wiki.mozilla.org/Buildbot/Talos/Tests Talos tests] locally. You can run suites of tests or individual tests on their own. |
|
| |
|
| == I have a patch for Talos, what tests to I run? ==
| | For example, to run the 'chromez' test suite: |
|
| |
|
| If you are making changes to talos obviously running the tests locally will be the first step. The next logical step is to run the tests on Try server (try: -b o -p all -u none -t all).
| | ./mach talos-test --suite chromez |
|
| |
|
| === Testing Locally ===
| | To run a single test (the 'tart' test): |
|
| |
|
| Testing locally involves running some subset of the Talos tests on
| | ./mach talos-test --activeTests tart |
| desktop and possibly mobile. Obviously not all permutations of tests
| |
| and the ways of running them can be tried, so common sense should be
| |
| used as to what is run. You may also want to run Talos' internal
| |
| unittests: http://hg.mozilla.org/build/talos/file/tip/tests
| |
|
| |
|
| You should tailor your choice of tests to pick those that cover what
| | To run two single tests, use the ':' to separate the test list: |
| you've changed programmatically, but in general you should probably
| |
| run at least one startup test and one pageloader test. A good
| |
| baseline might be:
| |
|
| |
|
| # refer to [[#Running_locally_-_Source_Code|running locally]] for more details
| | ./mach talos-test --activeTests tart:damp |
| talos -n -d -a ts:tsvg -e `which firefox` --develop --datazilla-url output.json --mozAfterPaint
| |
|
| |
|
| === Testing on Try Server ===
| | Certain talos suites/tests require external pagesets to be downloaded. If pagesets are required, they will be downloaded and extracted automatically when running talos via mach. To disable this automatic pageset downloading, add the 'no-download' argument: |
|
| |
|
| There are three scenarios that are needed for testing talos on try server:
| | ./mach talos-test --suite g1 --no-download |
| * Firefox Developer: testing your Firefox changes for performance issues
| |
| * Talos Validation: testing your already reviewed and landed patches to the Talos toolchain in production
| |
| * Talos Developer: testing patches to Talos which have not been reviewed or landed
| |
|
| |
|
| For the purposes of Talos development we will ignore the Firefox Developer scenario and discuss the two scenarios regarding Talos toolchain development. In fact, the process is similar for both, lets get started:
| | To see a list of available talos suites: |
|
| |
|
| First off, you need to edit [[http://dxr.mozilla.org/mozilla-central/source/testing/talos/talos.json?from=talos.json&case=true#1 testing/talos/talos.json]] in your Firefox tree, what is of most importance is:
| | ./mach talos-test --print-suites |
| "talos.zip": { | |
| "url": "http://talos-bundles.pvt.build.mozilla.org/zips/talos.49b74c08dad4.zip",
| |
| "path": ""
| |
| },
| |
| "global": {
| |
| "talos_repo": "https://hg.mozilla.org/build/talos",
| |
| "talos_revision": "5e18c3e56875"
| |
| }
| |
|
| |
|
| Android 4.0 tests use "talos.zip:, and all desktop platforms use the "global" section. You can see in global there are 2 fields, "talos_repo" and "talos_revision".
| | To see a list of available talos tests: |
|
| |
|
| In the Talos Validation scenario, you only need to change "talos_revision" to be the revision of [[http://hg.mozilla.org/build/talos talos]] that your change is.
| | ./mach talos-test --print-tests |
|
| |
|
| For the Talos Developer scenario, you would edit "global" to look like this:
| | Further help running talos via mach: |
| "global": {
| |
| "talos_repo": "http://hg.mozilla.org/users/jmaher_mozilla.com/tpain",
| |
| "talos_revision": "default"
| |
| }
| |
|
| |
|
| There are 2 key points here:
| | ./mach talos-test --help |
| * the repo is pointing to a [[https://developer.mozilla.org/en-US/docs/Creating_Mercurial_User_Repositories user repo]] which contains a full copy of the talos repo and all local changes made
| |
| * the revision is pointing at "default"
| |
|
| |
|
| ''NOTE: there is a trick with using "default" as the revision, it allows you to check in changes and then retrigger a job on try. This avoids the need to push a new revision to try and wait for a build.'' | | '''Tip:''' Clear the operating system cache between runs, in order to significantly improve the consistency of the results. For example, on linux: |
|
| |
|
| Finally if you are validating a push or adjusting something that should or will affect Android Talos, then you need to [[http://hg.mozilla.org/build/talos/file/5e18c3e56875/create_talos_zip.py create a new talos.zip]], ftp that to a people.mozilla.org directory, and then edit the talos.zip section of talos.json to reference it.
| | find . | xargs touch |
|
| |
|
| Once you have talos.json changed, you just need to select the proper [[http://trychooser.pub.build.mozilla.org/ try chooser]] syntax.
| | == Try Server == |
|
| |
|
| == Are my numbers ok ==
| | To run talos tests on the try server, you just simply push to try as normal, specifying that you want to run Talos tests. It is highly recommended that you use [http://trychooser.pub.build.mozilla.org/ trychooser] to build your try server syntax (helpful hint for using trychooser: in the list of talos suites available you can hover over any one of the suites, and a list of tests included in that suite will be displayed). |
|
| |
|
| The best way to answer this question is to push to try server and compare the reported numbers from the logs (use Treeherder as a log parser) and compare that with the [[http://graphs.mozilla.org/ graph server]]. I recommend using Treeherder to open the link to the graphs.
| | To run all of the talos tests as part of your try push, just add "-t all" to the end of your try syntax. To run a single talos suite as part of your try push, just add "-t <name of suite>" to your try syntax (i.e. "-t g1"). |
|
| |
|
| If you are planning on landing on mozilla-central, look at tests from mozilla-central. Be aware of PGO vs Non PGO and Chrome vs Non Chrome. Treeherder makes this a moderately pain free process (i.e. about 30 minutes). This is one of the big problems we are solving with datazilla.
| | To turn on [https://wiki.mozilla.org/Buildbot/Talos/Profiling profiling] as part of your talos tests on try, add "mozharness: --spsProfile" to the end of the try syntax (i.e. "-t g1 mozharness: --spsProfile"). There is an option to do this in trychooser. |
|
| |
|
| == Using try server ==
| | When viewing your talos try server results on Treeherder, look for the group names 'T' and 'T-e10s'. Click on the green symbol corresponding to the talos suite name, and then click on the 'Performance' tab below to see the test results. |
|
| |
|
| If you have access to generate try builds you can also have performance tests run against a custom version of talos. The performance results will be generated on the same machines that generate the talos results for all check-ins on all branches. This involves a few steps: | | If you're curious about the hardware we use, here is a [https://wiki.mozilla.org/Buildbot/Talos/Misc#Hardware_Profile_of_machines_used_in_automation description of the specs]. You can also refer to the [https://wiki.mozilla.org/ReferencePlatforms#Build_Reference_Platforms test reference platforms] documentation. |
|
| |
|
| === Testing changes to talos properly === | | == Are my Talos Results OK? == |
| * Create a [https://developer.mozilla.org/en-US/docs/Creating_Mercurial_User_Repositories?redirectlocale=en-US&redirectslug=Publishing_Mercurial_Clones user repo] with talos, commit and push your changes
| |
| ** If you are modifying Android specific tests, create_talos_zip.py from the root of your talos directory and upload the file somewhere that the build system can find it (e.g. http://people.mozilla.org/~wlachance/talos.zip)
| |
| * Check out a copy of Mozilla central
| |
| * Modify the file "testing/talos/talos.json" to point to your user repo and put the changeset as 'default'.
| |
| ** For android, change the location of talos.zip
| |
|
| |
|
| === Pushing to try ===
| | The best way to answer this question is to push to try server and compare the performance of your push against a baseline, using Perfherder (it is recommended hitting retrigger a few times to get at least 5 data points for each test). After pushing to try with talos tests enabled, you will receive an email with a link to Perfherder where you can compare your results. Be aware of the options: e.g. pgo vs non-pgo or e10s vs non-e10s. |
| * [https://wiki.mozilla.org/ReleaseEngineering/TryServer#How_to_push_to_try Push] this change to try server using the right syntax (you can use TryChooser to help with this: recommended is to test talos thoroughly, but standard unit tests can be skipped)
| |
| * Try server will give you results for a single data point (I recommend hitting retrigger a few times to get at least 3 data points for each test). Graph server is not very useful for viewing your try push data, we have two options for aiding in this effort:
| |
| ** [[http://hg.mozilla.org/build/talos/file/tip/talos/compare.py compare.py]]: clone talos, setup the virtualenv, run "python compare.py --help", normally I run "python compare.py --revision <rev>" where <rev> is the revision of my try push
| |
| ** [[http://compare-talos.mattn.ca/ compare-talos]]: Browse to the [http://hg.mozilla.org/build/talos/file/tip/talos/compare.py url], enter your try server revision, then enter a revision of the branch you are comparing to, you can then browser the delta for each summary and individual test page.
| |
|
| |
|
| A bit more information can be found in this [http://armenzg.blogspot.com/2011/12/taloszip-talosjson-and-you.html blog post] from armenzg.
| | = Hacking on Talos Itself = |
|
| |
|
| == Running locally - Source Code == | | == Dev Environment == |
|
| |
|
| * Note: you must first have setuptools installed on your system
| | Prerequisites: |
| * apt-get install python-setuptools # linux
| |
| * https://pypi.python.org/pypi/setuptools # download and install for all operating systems
| |
|
| |
|
| * you also need virtualenv installed on your system
| | Local clone of mozilla-central and a successful [https://developer.mozilla.org/en-US/docs/Simple_Firefox_build mozilla-central Firefox build] completed. |
| * pip install virtualenv # http://www.virtualenv.org/en/latest/virtualenv.html#installation
| |
|
| |
|
| For the majority of the tests, we include test files and tools out of the box. We need to do these things:
| | On Linux, setuptools installed: |
|
| |
|
| * '''clone talos:'''
| | apt-get install python-setuptools |
| hg clone http://hg.mozilla.org/build/talos
| |
|
| |
|
| * '''run the install script'''
| | Not required on Windows. For other operating systems see [https://pypi.python.org/pypi/setuptools Setuptools]. |
| cd talos
| | |
| python INSTALL.py
| | On Linux, virtualenv installed: |
| # (Ignore errors like "fatal error: 'yaml.h' file not found. To get rid of this error message you can download and install PyYAML from http://pyyaml.org/wiki/PyYAML but it is not necessary.
| |
|
| |
|
| ** creates a [https://developer.mozilla.org/en/Python/Virtualenv Virtualenv] in the same directory as 'INSTALL.py'
| | pip install virtualenv |
| ** installs the talos [https://developer.mozilla.org/en/Python#Python_packaging python package] into the virtualenv, including its [https://wiki.mozilla.org/Auto-tools/Projects/MozBase MozBase dependencies] via 'python setup.py develop'
| |
| ** ''(alternatively, you can perform these steps yourself)''
| |
| * '''activate the virtualenv:'''
| |
| ''(on windows):''
| |
| source Scripts\activate
| |
|
| |
|
| ''(on osx/linux)'':
| | Not required on Windows. For other operating systems see [http://www.virtualenv.org/en/latest/virtualenv.html#installation Virtualenv]. |
| . bin/activate
| |
|
| |
|
| * unpack a copy of firefox somewhere (for this example, we'll use `which firefox` as if firefox was on your path)
| | Setup talos (linux): |
| * '''setup a webserver''' if you're not using the '--develop' flag (WE STRONGLY RECOMMEND USING THE --develop FLAG)
| |
| ** setup apache or similar webserver to have http://localhost -> the talos subdirectory of the talos checkout
| |
| ** alternatively you can use the --develop flag to PerfConfigurator which configures to use a python webserver, [https://github.com/mozilla/mozbase/tree/master/mozhttpd mozhttpd], as shown below
| |
|
| |
|
| * '''run tests:'''
| | cd mozilla-central/testing/talos |
| talos -n -d --develop --executablePath pathtofirefox --activeTests ts --output ts_desktop.yml
| | virtualenv . |
| # Note: this will open and close the browser real fast 21 times
| | . bin/activate |
| ** --develop indicates to run in develop mode and to set up a webserver for you
| | python setup.py develop |
| ** --executablePath tells Talos where the firefox installation we want to run is located
| |
| *** we have pathtofirefox as an example, you can use '~/mozilla/objdir/dist/bin/firefox' or whatever the full path is to your firefox executable that will be running the tests.
| |
| ** --activeTests is a list of tests we want to run separated by ':'. In this example, we are running the startup test, ts.
| |
| ** --output is the new config file we want to generate
| |
| You can use `talos --help` to get a complete list of options
| |
|
| |
|
| '''Tip: Run ''find . | xargs touch'' to clear operating system caches between talos runs. This significantly improves the consistency of the results.'''
| | Setup talos (Windows with mozilla-build): |
|
| |
|
| If you're looking to run remote talos, instructions are at: https://wiki.mozilla.org/Mobile/Fennec/Android#talos
| | cd mozilla-central/testing/talos |
| | python INSTALL.py # only required once or after updating mozilla-build |
| | source Scripts/activate # you can also use '.' (dot) instead of 'source' |
|
| |
|
| ''talos --print-tests'' may be used to display the descriptions and run-time parameters of tests
| | == Testing your Talos Patch Locally == |
|
| |
|
| We do not include the tp5 or similar pagesets for legal restrictions.
| | Testing locally involves running some subset of the Talos tests on |
| | desktop and possibly mobile. Obviously not all permutations of tests |
| | and the ways of running them can be tried, so common sense should be |
| | used as to what is run. You may also want to run Talos' internal |
| | unittests: http://hg.mozilla.org/build/talos/file/tip/tests |
|
| |
|
| Talos will refuse to run if you have an open browser.
| | You should tailor your choice of tests to pick those that cover what |
| | | you've changed programmatically, but in general you should probably |
| If you want to load an extension while running Talos, you want a few more command line arguments:
| | run at least one startup test and one pageloader test. A good |
| | | baseline might be the 'tresize' and 'tsvgx' test suites. |
| talos -n -d --executablePath=../firefox/firefox --sampleConfig=sample.config --activeTests=ts --extension=mozmill-1.5.2-sb+tb+fx+sm.xpi --addonID=mozmill@mozilla.com --output=my.config
| |
| | |
| * --extension is the file of the XPI we want to install in the profile
| |
| * --addonID is the ID of the addon to install
| |
| | |
| == How Talos is Run in Production ==
| |
| | |
| * ''(TODO: about mozbase dependencies and create_talos.zip (elsewhere in the wiki? if so, link to from here))''
| |
| | |
| * buildbot constructs commands to launch [http://hg.mozilla.org/build/talos/file/tip/talos/PerfConfigurator.py PerfConfigurator] from a config file: http://hg.mozilla.org/build/buildbot-configs/raw-file/tip/mozilla-tests/config.py ; there are a number of suites, each of which may contain multiple tests
| |
| | |
| * a slave invokes [http://hg.mozilla.org/build/talos/file/tip/talos/run_tests.py run_tests.py] on the generated [http://www.yaml.org/ YAML] Talos configuration
| |
|
| |
|
| * Talos will run the tests and measured results which are uploaded to [http://graphs.mozilla.org/ graphserver] (after being suitably averaged per-page for Pageloader tests)
| | Run the tests from mozilla-central/testing/talos (with the venv activated, as noted above): |
| ** (Talos also uploads raw results to [https://github.com/mozilla/datazilla datazilla])
| |
|
| |
|
| * the graphserver performs averaging across the pageset (for Pageloader tests) or across cycles (for Startup tests) and returns a number via HTTP to Talos which is then printed to the log
| | talos --develop --executablePath pathtofirefox --activeTests tart |
|
| |
|
| * [https://treeherder.mozilla.org/ Treeherder] receives the name of the suite from buildbot. These are correlated to Treeherder letters via [https://github.com/mozilla/treeherder-service/blob/master/treeherder/etl/buildbot.py]. The computed results from graphserver are scraped from the log and displayed when a Treeherder suite letter is clicked.
| | The '--develop' option indicates to run in develop mode and store the results in local.json + local.out. |
|
| |
|
| === The Test Machines ===
| | The '--executablePath' option tells Talos where the firefox installation we want to run is located (i.e. '~/mozilla/objdir/dist/bin/firefox' or whatever the full path is to your firefox executable that will be running the tests). |
|
| |
|
| All talos test are run on a pool of 2.26 Ghz Intel Core 2 Duo, 2Gb 1067 MHz DDR3 mac minis.
| | The '--activeTests' argument provides is a list of tests we want to run (if running multiple tests, separate each test name with ':'). |
|
| |
|
| The machines are imaged to comply with the [https://wiki.mozilla.org/ReferencePlatforms#Build_Reference_Platforms Test Reference Platforms].
| | For a list of complete options: |
|
| |
|
| === Running Tp the Automation Way ===
| | talos --help |
| In our automation, we run beneath many more restrictions than normal users. One of those restrictions is that our automation machines are walled off from the real-world networks. Because of this, and because we want to test pure page-loading and rendering time of Firefox, we serve the pages from localhost using [http://httpd.apache.org/ Apache] thus eliminating all network latency and uncertainty. You've probably noticed this if you looked at the talos/page_load_test/tp4.manifest.
| |
|
| |
|
| To do this, we construct full downloads of sites in our manifest and they are placed on the automation slave at run time. Because we cannot at this time distribute our full page load test set, I'll walk through how these are set up and show you how to make your own. Note that our next version of the page load set will be distributable, so soon this won't be an issue.
| | == Testing your Talos Patch on Try Server == |
|
| |
|
| In the meantime, here's the instructions:
| | Just push your talos patch to the try server. See [https://wiki.mozilla.org/Buildbot/Talos/Running#Try_Server the Try Server section] above. |
| <ol>
| |
| <li>Use [http://people.mozilla.org/~ctalbert/talos/makepagebundle.sh this script] or, you use the following wget command to fetch a page and everything it links to in order to have a complete page for offline use: <pre>wget -p -k -H -E -erobots=off --no-check-certificate -U "Mozilla/5.0 (firefox)" --restrict-file-names=windows --restrict-file-names=nocontrol $URL -o outputlog.txt</pre>
| |
| </li>
| |
| <li>Once you have a cache of pages, install Apache: <pre>sudo apt-get install apache2</pre></li>
| |
| <li>Copy your page into the proper location for Apache to serve it. Note that I like to create a page_load_test directory to separate talos from anything else on the webserver. So with Apache defaults, that's something like: <pre>mkdir /var/www/page_load_test; cp -R <dir> /var/www/page_load_test/.</pre></li>
| |
| <li>Now, add the local URL into your manifest: <pre>http://localhost/page_load_test/<dir></pre></li>
| |
| <li>Run the tp tests as above, pointing the config file at your manifest.
| |
| </li>
| |
| </ol>
| |