TestEngineering/Performance/Raptor/Browsertime: Difference between revisions
(Added alternate chromedriver sync method) |
(Added information on how to retrigger visual metric browsertime tasks.) |
||
| Line 75: | Line 75: | ||
./mach try fuzzy --browsertime -q "'test-android-hw-g5-7-0-arm7-api-16/opt-raptor-tp6m-1-geckoview-e10s" | ./mach try fuzzy --browsertime -q "'test-android-hw-g5-7-0-arm7-api-16/opt-raptor-tp6m-1-geckoview-e10s" | ||
=== Retriggering Browsertime Visual Metrics Tasks === | |||
After running Browsertime tests with visual metrics enabled on try you will notice that retriggering the tasks (along with the vismet tasks) is not straightforward. To retrigger them correctly, you will have to do the following: | |||
* Select a browsertime test task (not a vismet task) that you want to retrigger. | |||
* In the Treeherder pop-up menu, select the <tt>...</tt> and click on <tt>Custom Action</tt>. | |||
* Pick the <tt>retrigger</tt> action if it wasn't already selected. | |||
* Set <tt>downstream</tt> to <tt>true</tt>. | |||
* Enter the number of times it should be retriggered in <tt>times</tt>. | |||
* Retrigger! | |||
See [https://bugzilla.mozilla.org/show_bug.cgi?id=1602893 bug 1602893] for more information on this issue. | |||
=== Upgrading Browsertime In-Tree === | === Upgrading Browsertime In-Tree === | ||
Revision as of 21:46, 14 January 2020
Browsertime is a harness for running performance tests, similar to Mozilla's Raptor testing framework. Browsertime is written in Node.js and uses Selenium WebDriver to drive multiple browsers including Chrome, Chrome for Android, Firefox, and (pending the resolution of bug 1525126 and similar tickets) Firefox for Android and GeckoView-based vehicles.
Source code:
- Our current (vendored) Browsertime uses the Mozilla fork of the canonical repo
- In-tree: https://searchfox.org/mozilla-central/source/tools/browsertime and https://searchfox.org/mozilla-central/source/taskcluster/scripts/misc/browsertime.sh
Page-load tests
Running Locally
Prerequisites
- A local mozilla repository clone with a successful Firefox build completed
Setup
- Run
./mach browsertime --setup - To check your setup, run
./mach browsertime --check, which will output something like:
ffmpeg: OK convert: OK compare: OK Pillow: OK SSIM: OK
- If
ffmpegis listed as FAIL, you might want to try this:
cd ~/.mozbuild/browsertime/ffmpeg-4.1.1-macos64-static/bin chmod +x ffmpeg ffplay ffprobe}}
Now, try re-running ./mach browsertime --check, with ffmpeg hopefully fixed
- For other issues, see if
./mach browsertime --setup --clobberfixes it, or deleting the ~/.mozbuild/browsertime folder and running the aforementioned command.
If convert and compare are also FAIL bugs which might further help are bug 1559168, bug 1559727, and bug 1574964, for starters. If none of the bugs are related to the issue, please file a bug Testing :: Performance.
- If you plan on running Browsertime on Android, your Android device must already be set up (see more below in the Android section)
Running on Firefox Desktop
Page-load tests
- Browsertime-"native":
./mach browsertime https://www.sitespeed.io --firefox.binaryPath '/Users/{userdir}/moz_src/mozilla-unified/obj-x86_64-apple-darwin18.7.0/dist/Nightly.app/Contents/MacOS/firefox'
- Raptor-wrapped:
./mach raptor --browsertime -t raptor-tp6-google-firefox
Benchmark tests
- Raptor-wrapped:
./mach raptor -t raptor-speedometer --browsertime
Running on Android
.
- Running on Fenix/Firefox Preview
./mach browsertime --android --browser firefox --firefox.android.package org.mozilla.fenix.debug --firefox.android.activity org.mozilla.fenix.IntentReceiverActivity https://www.sitespeed.io
- Running on the GeckoView Example app
./mach browsertime --android --browser firefox https://www.sitespeed.io
- Running on Reference Browser
.
Running on Google Chrome
Chrome releases are tied to a specific version of ChromeDriver -- you will need to ensure the two are aligned.
There are two ways of doing this:
1. Download the ChromeDriver that matches the chrome you wish to run from https://chromedriver.chromium.org/ and specify the path:
./mach browsertime https://www.sitespeed.io -b chrome --chrome.chromedriverPath <PATH/TO/VERSIONED/CHROMEDRIVER>
2. Upgrade the ChromeDriver version in tools/browsertime/package-lock.json (see https://www.npmjs.com/package/@sitespeed.io/chromedriver for versions).
Run npm install.
Launch browsertime as follows:
./mach browsertime https://www.sitespeed.io -b chrome
More Examples
Running Browsertime on Try
Include the --browsertime flag to mach try fuzzy. As of bug 1566174 this will arrange for the jobs to include dependencies for Browsertime (Node.js, Browsertime node_modules, geckodriver, chromedriver, etc) and for the Raptor harness to run the given pageload test using Browsertime.
Not all Raptor tasks use Browsertime meaningfully. Right now, only cold pageload tests are supported. See bug 1577905 for warm pageload tests, for example.
For example:
./mach try fuzzy --browsertime -q "'test-android-hw-g5-7-0-arm7-api-16/opt-raptor-tp6m-1-geckoview-e10s"
Retriggering Browsertime Visual Metrics Tasks
After running Browsertime tests with visual metrics enabled on try you will notice that retriggering the tasks (along with the vismet tasks) is not straightforward. To retrigger them correctly, you will have to do the following:
- Select a browsertime test task (not a vismet task) that you want to retrigger.
- In the Treeherder pop-up menu, select the ... and click on Custom Action.
- Pick the retrigger action if it wasn't already selected.
- Set downstream to true.
- Enter the number of times it should be retriggered in times.
- Retrigger!
See bug 1602893 for more information on this issue.
Upgrading Browsertime In-Tree
To upgrade the browsertime version used in-tree follow the changes in bug 1592400.
To perform the upgrade, you need to make a change to tools/browsertime/package.json where you need to change the commit of the browsertime tarball. After making the change, upgrade browsertime locally, then include the tools/browsertime/package-lock.json changes in the upgrade patch. There might not be any changes to it. In that case, run npm install in tools/browsertime/ to update it, you should at the least see a change in the "integrity" and "version" fields.
To test the upgrade, first run ./mach browsertime https://sitespeed.io and compare the before/after results. Ensure that --video still works on the upgraded version as well. If all that works, then run all browsertime tests available in production. If everything looks normal, or expected, then the upgrade is good to land.
Bugs
60 Total; 5 Open (8.33%); 54 Resolved (90%); 1 Verified (1.67%);