TestEngineering/Performance/Raptor/Browsertime: Difference between revisions
(→Running on Google Chrome: Add raptor-browsertime usage.) |
(→Upgrading Browsertime In-Tree: Update information here.) |
||
| Line 95: | Line 95: | ||
=== Upgrading Browsertime In-Tree === | === Upgrading Browsertime In-Tree === | ||
To upgrade the browsertime version used in-tree | To upgrade the browsertime version used in-tree you can run, then commit the changes made to <code>package.json</code> and <code>package-lock.json</code>: | ||
./mach browsertime --update-upstream-url <TARBALL-URL> | |||
Here is a sample URL that we can update to: https://github.com/sitespeedio/browsertime/tarball/89771a1d6be54114db190427dbc281582cba3d47 | |||
To test the upgrade, | To test the upgrade, run a raptor test locally (with and without visual-metrics <code>--browsertime-visualmetrics</code> if possible) and test it on try with at least one test on desktop and mobile. | ||
=== Finding the Geckodriver Being Used === | === Finding the Geckodriver Being Used === | ||
Revision as of 17:19, 11 January 2021
Browsertime is a harness for running performance tests, similar to Mozilla's Raptor testing framework. Browsertime is written in Node.js and uses Selenium WebDriver to drive multiple browsers including Chrome, Chrome for Android, Firefox, and (pending the resolution of bug 1525126 and similar tickets) Firefox for Android and GeckoView-based vehicles.
Source code:
- Our current Browsertime version uses the canonical repo.
- In-tree: https://searchfox.org/mozilla-central/source/tools/browsertime and https://searchfox.org/mozilla-central/source/taskcluster/scripts/misc/browsertime.sh
Running Locally
Prerequisites
- A local mozilla repository clone with a successful Firefox build completed
Setup
Note that if you are running Raptor-Browsertime then it will get installed automatically and will also update itself once bug 1686064 lands.
- Run
./mach browsertime --setup
- To check your setup, run
./mach browsertime --check, which will output something like:
ffmpeg: OK convert: OK compare: OK Pillow: OK SSIM: OK
- If
ffmpegis listed as FAIL, you might want to try this:
cd ~/.mozbuild/browsertime/ffmpeg-4.1.1-macos64-static/bin chmod +x ffmpeg ffplay ffprobe}}
Now, try re-running ./mach browsertime --check, with ffmpeg hopefully fixed
- For other issues, see if
./mach browsertime --setup --clobberfixes it, or deleting the ~/.mozbuild/browsertime folder and running the aforementioned command.
- If you aren't running visual metrics, then failures in
PillowandSSIMcan be ignored.
If convert and compare are also FAIL bugs which might further help are bug 1559168, bug 1559727, and bug 1574964, for starters. If none of the bugs are related to the issue, please file a bug Testing :: Raptor.
- If you plan on running Browsertime on Android, your Android device must already be set up (see more below in the Android section)
Running on Firefox Desktop
Page-load tests
There are two ways to run performance tests through browsertime listed below. Note that `./mach browsertime` should not be used when debugging performance issues with profiles as it does not do symbolication.
- Browsertime-"native":
./mach browsertime https://www.sitespeed.io --firefox.binaryPath '/Users/{userdir}/moz_src/mozilla-unified/obj-x86_64-apple-darwin18.7.0/dist/Nightly.app/Contents/MacOS/firefox'
- Raptor-wrapped:
./mach raptor --browsertime -t raptor-tp6-google-firefox
Benchmark tests
- Raptor-wrapped:
./mach raptor -t raptor-speedometer --browsertime
Running on Android
Running on Raptor-Browsertime (recommended):
- Running on Fenix
./mach raptor --browsertime -t amazon --no-conditioned-profile --app fenix --binary org.mozilla.fenix
- Running on Geckoview
./mach raptor --browsertime -t amazon --no-conditioned-profile --app geckoview --binary org.mozilla.geckoview_example
Running on vanilla Browsertime:
- Running on Fenix/Firefox Preview
./mach browsertime --android --browser firefox --firefox.android.package org.mozilla.fenix.debug --firefox.android.activity org.mozilla.fenix.IntentReceiverActivity https://www.sitespeed.io
- Running on the GeckoView Example app
./mach browsertime --android --browser firefox https://www.sitespeed.io
Running on Google Chrome
Chrome releases are tied to a specific version of ChromeDriver -- you will need to ensure the two are aligned.
There are two ways of doing this:
1. Download the ChromeDriver that matches the chrome you wish to run from https://chromedriver.chromium.org/ and specify the path:
./mach browsertime https://www.sitespeed.io -b chrome --chrome.chromedriverPath <PATH/TO/VERSIONED/CHROMEDRIVER>
2. Upgrade the ChromeDriver version in tools/browsertime/package-lock.json (see https://www.npmjs.com/package/@sitespeed.io/chromedriver for versions).
Run npm install.
Launch vanilla Browsertime as follows:
./mach browsertime https://www.sitespeed.io -b chrome
Or for Raptor-Browsertime (use chrome for desktop, and chrome-m for mobile):
./mach raptor --browsertime -t amazon --app chrome --no-conditioned-profile --browsertime-chromedriver <PATH/TO/CHROMEDRIVER>
More Examples
Running Browsertime on Try
Include the --browsertime flag to mach try fuzzy. As of bug 1566174 this will arrange for the jobs to include dependencies for Browsertime (Node.js, Browsertime node_modules, geckodriver, chromedriver, etc) and for the Raptor harness to run the given pageload test using Browsertime.
Not all Raptor tasks use Browsertime meaningfully. Right now, only cold pageload tests are supported. See bug 1577905 for warm pageload tests, for example.
For example:
./mach try fuzzy --browsertime -q "'test-android-hw-g5-7-0-arm7-api-16/opt-raptor-tp6m-1-geckoview-e10s"
Retriggering Browsertime Visual Metrics Tasks
For details on retriggering/backfilling visual metrics tasks see VisualMetrics.
Upgrading Browsertime In-Tree
To upgrade the browsertime version used in-tree you can run, then commit the changes made to package.json and package-lock.json:
./mach browsertime --update-upstream-url <TARBALL-URL>
Here is a sample URL that we can update to: https://github.com/sitespeedio/browsertime/tarball/89771a1d6be54114db190427dbc281582cba3d47
To test the upgrade, run a raptor test locally (with and without visual-metrics --browsertime-visualmetrics if possible) and test it on try with at least one test on desktop and mobile.
Finding the Geckodriver Being Used
If you're looking for the latest geckodriver being used there are two ways:
- Find the latest one from here: https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=geckodriver
- Alternatively, if you're trying to figure out which geckodriver a given CI task is using, you can click on the browsertime task in treeherder, and then click on the `Task` id in the bottom left of the pop-up interface. Then in the window that opens up, click on `See more` in the task details tab on the left, this will show you the dependent tasks with the latest toolchain-geckodriver being used. There's an Artifacts drop down on the right hand side for the toolchain-geckodriver task that you can find the latest geckodriver in.
Comparing Before/After Browsertime Videos
We have some scripts that can produce side-by-side comparison videos for you of the worst pairing of videos. You can find the script here: https://github.com/gmierz/moz-current-tests#browsertime-side-by-side-video-comparisons
Once the side-by-side comparison is produced, the video on the left is the old/base video, and the video on the right is the new video.
Bugs
60 Total; 5 Open (8.33%); 54 Resolved (90%); 1 Verified (1.67%);