342
edits
TestEngineering/Performance/Raptor/Browsertime: Difference between revisions
TestEngineering/Performance/Raptor/Browsertime (view source)
Revision as of 13:46, 17 June 2021
, 17 June 2021Redirected page to TestEngineering/Performance/Raptor#WebExtension
(→Running on Android: Add raptor-browsertime information.) |
Alexandrui (talk | contribs) (Redirected page to TestEngineering/Performance/Raptor#WebExtension) |
||
| (12 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
Browsertime is a harness for running performance tests, similar to Mozilla's Raptor testing framework. Browsertime is written in Node.js and uses Selenium WebDriver to drive multiple browsers including Chrome, Chrome for Android, Firefox, and | #REDIRECT [[TestEngineering/Performance/Raptor#WebExtension]] | ||
Browsertime is a harness for running performance tests, similar to Mozilla's Raptor testing framework. Browsertime is written in Node.js and uses Selenium WebDriver to drive multiple browsers including Chrome, Chrome for Android, Firefox, and Firefox for Android and GeckoView-based vehicles. | |||
Source code: | Source code: | ||
| Line 10: | Line 12: | ||
=== Setup === | === Setup === | ||
Note that if you are running Raptor-Browsertime then it will get installed automatically and | '''Note that if you are running Raptor-Browsertime then it will get installed automatically and also updates itself.''' | ||
* Run <code>./mach browsertime --setup</code> | * Run <code>./mach browsertime --setup</code> | ||
| Line 42: | Line 44: | ||
There are two ways to run performance tests through browsertime listed below. '''Note that `./mach browsertime` should not be used when debugging performance issues with profiles as it does not do symbolication.''' | There are two ways to run performance tests through browsertime listed below. '''Note that `./mach browsertime` should not be used when debugging performance issues with profiles as it does not do symbolication.''' | ||
* Raptor-Browsertime (recommended):<br> | |||
./mach raptor --browsertime -t google-search | |||
* Browsertime-"native":<br> | * Browsertime-"native":<br> | ||
./mach browsertime https://www.sitespeed.io --firefox.binaryPath '/Users/{userdir}/moz_src/mozilla-unified/obj-x86_64-apple-darwin18.7.0/dist/Nightly.app/Contents/MacOS/firefox' | ./mach browsertime https://www.sitespeed.io --firefox.binaryPath '/Users/{userdir}/moz_src/mozilla-unified/obj-x86_64-apple-darwin18.7.0/dist/Nightly.app/Contents/MacOS/firefox' | ||
=== Benchmark tests === | === Benchmark tests === | ||
| Line 54: | Line 56: | ||
Running on Raptor-Browsertime (recommended): | Running on Raptor-Browsertime (recommended): | ||
* Running on Fenix | * Running on Fenix | ||
./mach raptor --browsertime -t amazon | ./mach raptor --browsertime -t amazon --app fenix --binary org.mozilla.fenix | ||
* Running on Geckoview | * Running on Geckoview | ||
./mach raptor --browsertime -t amazon | ./mach raptor --browsertime -t amazon --app geckoview --binary org.mozilla.geckoview_example | ||
Running on vanilla Browsertime: | Running on vanilla Browsertime: | ||
| Line 74: | Line 76: | ||
2. Upgrade the ChromeDriver version in <code>tools/browsertime/package-lock.json </code> (see https://www.npmjs.com/package/@sitespeed.io/chromedriver for versions). | 2. Upgrade the ChromeDriver version in <code>tools/browsertime/package-lock.json </code> (see https://www.npmjs.com/package/@sitespeed.io/chromedriver for versions). | ||
Run <code>npm install</code>. | Run <code>npm install</code>. | ||
Launch | |||
Launch vanilla Browsertime as follows: | |||
./mach browsertime https://www.sitespeed.io -b chrome | ./mach browsertime https://www.sitespeed.io -b chrome | ||
Or for Raptor-Browsertime (use <code>chrome</code> for desktop, and <code>chrome-m</code> for mobile): | |||
./mach raptor --browsertime -t amazon --app chrome --browsertime-chromedriver <PATH/TO/CHROMEDRIVER> | |||
=== More Examples === | === More Examples === | ||
[https://github.com/mozilla/browsertime/tree/master/docs/examples Browsertime docs] | [https://github.com/mozilla/browsertime/tree/master/docs/examples Browsertime docs] | ||
=== Running Browsertime on Try === | === Running Browsertime on Try === | ||
You can run all of our browsertime pageload tests through <code>./mach try fuzzy --full</code>. We use chimera mode in these tests which means that both cold and warm pageload variants are running at the same time. | |||
For example: | For example: | ||
./mach try fuzzy | ./mach try fuzzy -q "'g5 'imdb 'geckoview 'vismet '-wr 'shippable" | ||
=== Retriggering Browsertime Visual Metrics Tasks === | === Retriggering Browsertime Visual Metrics Tasks === | ||
For details on | |||
You can retrigger Browsertime tasks just like you retrigger any other tasks from Treeherder (using the retrigger buttons, add-new-jobs, retrigger-multiple, etc.). | |||
When you retrigger the Browsertime test task, it will trigger a new vismet task as well. If you retrigger a Browsertime vismet task, then it will cause the test task to be retriggered and a new vismet task will be produced from there. This means that both of these tasks are treated as "one" task when it comes to retriggering them. | |||
There is only one path that still doesn't work for retriggering Browsertime tests and that's happens when you use `--rebuild X` in a try push submission. | |||
For details on how we previously retriggered visual metrics tasks see [[/VisualMetrics/]] (this will stay here for a few months just in case). | |||
=== Gecko Profiling with Browsertime === | |||
To run gecko profiling using Raptor-Browsertime you can add the <code>--gecko-profile</code> flag to any command and you will get profiles from the test (with the profiler page opening in the browser automatically). This method also performs symbolication for you. For example: | |||
./mach raptor --browsertime -t amazon --gecko-profile | |||
Note that vanilla Browsertime does support Gecko Profiling but '''it does not symbolicate the profiles''' so it is '''not recommended''' to use for debugging performance regressions/improvements. | |||
=== Upgrading Browsertime In-Tree === | === Upgrading Browsertime In-Tree === | ||
To upgrade the browsertime version used in-tree | To upgrade the browsertime version used in-tree you can run, then commit the changes made to <code>package.json</code> and <code>package-lock.json</code>: | ||
./mach browsertime --update-upstream-url <TARBALL-URL> | |||
Here is a sample URL that we can update to: https://github.com/sitespeedio/browsertime/tarball/89771a1d6be54114db190427dbc281582cba3d47 | |||
To test the upgrade, | To test the upgrade, run a raptor test locally (with and without visual-metrics <code>--browsertime-visualmetrics</code> if possible) and test it on try with at least one test on desktop and mobile. | ||
=== Finding the Geckodriver Being Used === | === Finding the Geckodriver Being Used === | ||
| Line 101: | Line 120: | ||
* Find the latest one from here: https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=geckodriver | * Find the latest one from here: https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=geckodriver | ||
* Alternatively, if you're trying to figure out which geckodriver a given CI task is using, you can click on the browsertime task in treeherder, and then click on the `Task` id in the bottom left of the pop-up interface. Then in the window that opens up, click on `See more` in the task details tab on the left, this will show you the dependent tasks with the latest toolchain-geckodriver being used. There's an Artifacts drop down on the right hand side for the toolchain-geckodriver task that you can find the latest geckodriver in. | * Alternatively, if you're trying to figure out which geckodriver a given CI task is using, you can click on the browsertime task in treeherder, and then click on the `Task` id in the bottom left of the pop-up interface. Then in the window that opens up, click on `See more` in the task details tab on the left, this will show you the dependent tasks with the latest toolchain-geckodriver being used. There's an Artifacts drop down on the right hand side for the toolchain-geckodriver task that you can find the latest geckodriver in. | ||
If you're trying to test Browsertime with a new geckodriver, you can do either of the following: | |||
* Request a new geckodriver build in your try run (i.e. through <code>./mach try fuzzy</code>). | |||
* Trigger a new geckodriver in a try push, then trigger the browsertime tests which will then use the newly built version in the try push. | |||
=== Comparing Before/After Browsertime Videos === | === Comparing Before/After Browsertime Videos === | ||