TestEngineering/Performance/Raptor/Browsertime: Difference between revisions
< TestEngineering | Performance | Raptor
Jump to navigation
Jump to search
No edit summary |
|||
| Line 1: | Line 1: | ||
Browsertime is a harness for running performance tests, similar to | Browsertime is a harness for running performance tests, similar to Mozilla's Raptor testing framework. Browsertime is written in Node.js and uses Selenium WebDriver to drive multiple browsers including Chrome, Chrome for Android, Firefox, and (pending the resolution of {{bug|1525126}} and similar tickets) Firefox for Android and GeckoView-based vehicles. | ||
Mozilla's Raptor testing framework. | |||
and uses Selenium WebDriver to drive multiple browsers including | |||
Chrome, Chrome for Android, Firefox, and (pending the resolution of | |||
and similar tickets) Firefox for Android and GeckoView-based vehicles. | |||
Source code: | Source code: | ||
| Line 16: | Line 11: | ||
=== Setup === | === Setup === | ||
* Run <code>./mach browsertime --setup</code> | * Run <code>./mach browsertime --setup</code> | ||
* To check your setup, run <code>./mach browsertime --check</code>, which will output something like: | * To check your setup, run <code>./mach browsertime --check</code>, which will output something like: | ||
ffmpeg: OK | ffmpeg: OK | ||
convert: OK | convert: OK | ||
compare: OK | compare: OK | ||
Pillow: OK | Pillow: OK | ||
SSIM: OK | SSIM: OK | ||
* If <code>ffmpeg</code> is listed as FAIL, you might want to try this: | |||
cd ~/.mozbuild/browsertime/ffmpeg-4.1.1-macos64-static/bin | |||
chmod +x ffmpeg ffplay ffprobe}} | |||
Now, try re-running <code>./mach browsertime --check</code>, with <code>ffmpeg</code> hopefully fixed | Now, try re-running <code>./mach browsertime --check</code>, with <code>ffmpeg</code> hopefully fixed | ||
If < | If <code>convert</code> and <code>compare</code> are also <code>FAIL</code> bugs which might further help are {{bug|1559168}}, {{bug|1559727}}, and {{bug|1574964}}, for starters. | ||
* If you plan on running Browsertime on Android, your Android device must already be set up (see more below in the Android section) | * If you plan on running Browsertime on Android, your Android device must already be set up (see more below in the Android section) | ||
Revision as of 13:44, 29 October 2019
Browsertime is a harness for running performance tests, similar to Mozilla's Raptor testing framework. Browsertime is written in Node.js and uses Selenium WebDriver to drive multiple browsers including Chrome, Chrome for Android, Firefox, and (pending the resolution of bug 1525126 and similar tickets) Firefox for Android and GeckoView-based vehicles.
Source code:
- Our current (vendored) Browsertime uses the Mozilla fork of the canonical repo
- In-tree: https://searchfox.org/mozilla-central/source/tools/browsertime and https://searchfox.org/mozilla-central/source/taskcluster/scripts/misc/browsertime.sh
Page-load tests
Running Locally
Prerequisites
- A local mozilla repository clone with a successful Firefox build completed
Setup
- Run
./mach browsertime --setup - To check your setup, run
./mach browsertime --check, which will output something like:
ffmpeg: OK convert: OK compare: OK Pillow: OK SSIM: OK
- If
ffmpegis listed as FAIL, you might want to try this:
cd ~/.mozbuild/browsertime/ffmpeg-4.1.1-macos64-static/bin chmod +x ffmpeg ffplay ffprobe}}
Now, try re-running ./mach browsertime --check, with ffmpeg hopefully fixed
If convert and compare are also FAIL bugs which might further help are bug 1559168, bug 1559727, and bug 1574964, for starters.
- If you plan on running Browsertime on Android, your Android device must already be set up (see more below in the Android section)
Running on Firefox Desktop
- Browsertime-"native":
./mach browsertime https://www.sitespeed.io --firefox.binaryPath '/Users/{userdir}/moz_src/mozilla-unified/obj-x86_64-apple-darwin18.7.0/dist/Nightly.app/Contents/MacOS/firefox'
- Raptor-wrapped:
./mach raptor --browsertime -t raptor-tp6-google-firefox
Running on Fennec
.
Running on Fenix/Firefox Preview
./mach browsertime --android --browser firefox --firefox.android.package org.mozilla.fenix.debug --firefox.android.activity org.mozilla.fenix.IntentReceiverActivity https://www.sitespeed.io
Running on the GeckoView Example app
./mach browsertime --android --browser firefox https://www.sitespeed.io
Running on Reference Browser
.
Running on Google Chrome
./mach browsertime https://www.sitespeed.io -b chrome
More Examples
Running Browsertime on Try
Include the --browsertime flag to mach try fuzzy. As of bug 1566174 this will arrange for the jobs to include dependencies for Browsertime (Node.js, Browsertime node_modules, geckodriver, chromedriver, etc) and for the Raptor harness to run the given pageload test using Browsertime.
Not all Raptor tasks use Browsertime meaningfully. Right now, only cold pageload tests are supported. See bug 1577905 for warm pageload tests, for example.
For example:
./mach try fuzzy --browsertime -q "'test-android-hw-g5-7-0-arm7-api-16/opt-raptor-tp6m-1-geckoview-e10s"
Bugs
60 Total; 5 Open (8.33%); 54 Resolved (90%); 1 Verified (1.67%);