TestEngineering/Performance/Raptor/Browsertime: Difference between revisions

Latest revision as of 13:46, 17 June 2021

Redirect to:

TestEngineering/Performance/Raptor#WebExtension

Browsertime is a harness for running performance tests, similar to Mozilla's Raptor testing framework. Browsertime is written in Node.js and uses Selenium WebDriver to drive multiple browsers including Chrome, Chrome for Android, Firefox, and Firefox for Android and GeckoView-based vehicles.

Source code:

Our current Browsertime version uses the canonical repo.
In-tree: https://searchfox.org/mozilla-central/source/tools/browsertime and https://searchfox.org/mozilla-central/source/taskcluster/scripts/misc/browsertime.sh

Running Locally

Prerequisites

A local mozilla repository clone with a successful Firefox build completed

Setup

Note that if you are running Raptor-Browsertime then it will get installed automatically and also updates itself.

Run ./mach browsertime --setup

To check your setup, run ./mach browsertime --check, which will output something like:

 ffmpeg:   OK 
 convert:  OK 
 compare:  OK
 Pillow:   OK
 SSIM:     OK

If ffmpeg is listed as FAIL, you might want to try this:

 cd ~/.mozbuild/browsertime/ffmpeg-4.1.1-macos64-static/bin
 chmod +x ffmpeg ffplay ffprobe}}

Now, try re-running ./mach browsertime --check, with ffmpeg hopefully fixed

For other issues, see if ./mach browsertime --setup --clobber fixes it, or deleting the ~/.mozbuild/browsertime folder and running the aforementioned command.

If you aren't running visual metrics, then failures in Pillow and SSIM can be ignored.

If convert and compare are also FAIL bugs which might further help are bug 1559168, bug 1559727, and bug 1574964, for starters. If none of the bugs are related to the issue, please file a bug Testing :: Raptor.

If you plan on running Browsertime on Android, your Android device must already be set up (see more below in the Android section)

Running on Firefox Desktop

Page-load tests

There are two ways to run performance tests through browsertime listed below. Note that `./mach browsertime` should not be used when debugging performance issues with profiles as it does not do symbolication.

Raptor-Browsertime (recommended):

./mach raptor --browsertime -t google-search

Browsertime-"native":

 ./mach browsertime https://www.sitespeed.io --firefox.binaryPath '/Users/{userdir}/moz_src/mozilla-unified/obj-x86_64-apple-darwin18.7.0/dist/Nightly.app/Contents/MacOS/firefox'

Benchmark tests

Raptor-wrapped:

./mach raptor -t raptor-speedometer --browsertime

Running on Android

Running on Raptor-Browsertime (recommended):

Running on Fenix

 ./mach raptor --browsertime -t amazon --app fenix --binary org.mozilla.fenix

Running on Geckoview

 ./mach raptor --browsertime -t amazon --app geckoview --binary org.mozilla.geckoview_example

Running on vanilla Browsertime:

Running on Fenix/Firefox Preview

 ./mach browsertime --android --browser firefox --firefox.android.package org.mozilla.fenix.debug --firefox.android.activity org.mozilla.fenix.IntentReceiverActivity https://www.sitespeed.io

Running on the GeckoView Example app

 ./mach browsertime --android --browser firefox https://www.sitespeed.io

Running on Google Chrome

Chrome releases are tied to a specific version of ChromeDriver -- you will need to ensure the two are aligned.

There are two ways of doing this:

1. Download the ChromeDriver that matches the chrome you wish to run from https://chromedriver.chromium.org/ and specify the path:

 ./mach browsertime https://www.sitespeed.io -b chrome --chrome.chromedriverPath <PATH/TO/VERSIONED/CHROMEDRIVER>

2. Upgrade the ChromeDriver version in tools/browsertime/package-lock.json (see https://www.npmjs.com/package/@sitespeed.io/chromedriver for versions). Run npm install.

Launch vanilla Browsertime as follows:

 ./mach browsertime https://www.sitespeed.io -b chrome

Or for Raptor-Browsertime (use chrome for desktop, and chrome-m for mobile):

 ./mach raptor --browsertime -t amazon --app chrome --browsertime-chromedriver <PATH/TO/CHROMEDRIVER>

More Examples

Browsertime docs

Running Browsertime on Try

You can run all of our browsertime pageload tests through ./mach try fuzzy --full. We use chimera mode in these tests which means that both cold and warm pageload variants are running at the same time.

For example:

 ./mach try fuzzy -q "'g5 'imdb 'geckoview 'vismet '-wr 'shippable"

Retriggering Browsertime Visual Metrics Tasks

You can retrigger Browsertime tasks just like you retrigger any other tasks from Treeherder (using the retrigger buttons, add-new-jobs, retrigger-multiple, etc.).

When you retrigger the Browsertime test task, it will trigger a new vismet task as well. If you retrigger a Browsertime vismet task, then it will cause the test task to be retriggered and a new vismet task will be produced from there. This means that both of these tasks are treated as "one" task when it comes to retriggering them.

There is only one path that still doesn't work for retriggering Browsertime tests and that's happens when you use `--rebuild X` in a try push submission.

For details on how we previously retriggered visual metrics tasks see VisualMetrics (this will stay here for a few months just in case).

Gecko Profiling with Browsertime

To run gecko profiling using Raptor-Browsertime you can add the --gecko-profile flag to any command and you will get profiles from the test (with the profiler page opening in the browser automatically). This method also performs symbolication for you. For example:

 ./mach raptor --browsertime -t amazon --gecko-profile

Note that vanilla Browsertime does support Gecko Profiling but it does not symbolicate the profiles so it is not recommended to use for debugging performance regressions/improvements.

Upgrading Browsertime In-Tree

To upgrade the browsertime version used in-tree you can run, then commit the changes made to package.json and package-lock.json:

 ./mach browsertime --update-upstream-url <TARBALL-URL>

Here is a sample URL that we can update to: https://github.com/sitespeedio/browsertime/tarball/89771a1d6be54114db190427dbc281582cba3d47

To test the upgrade, run a raptor test locally (with and without visual-metrics --browsertime-visualmetrics if possible) and test it on try with at least one test on desktop and mobile.

Finding the Geckodriver Being Used

If you're looking for the latest geckodriver being used there are two ways:

Find the latest one from here: https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=geckodriver
Alternatively, if you're trying to figure out which geckodriver a given CI task is using, you can click on the browsertime task in treeherder, and then click on the `Task` id in the bottom left of the pop-up interface. Then in the window that opens up, click on `See more` in the task details tab on the left, this will show you the dependent tasks with the latest toolchain-geckodriver being used. There's an Artifacts drop down on the right hand side for the toolchain-geckodriver task that you can find the latest geckodriver in.

If you're trying to test Browsertime with a new geckodriver, you can do either of the following:

Request a new geckodriver build in your try run (i.e. through ./mach try fuzzy).
Trigger a new geckodriver in a try push, then trigger the browsertime tests which will then use the newly built version in the try push.

Comparing Before/After Browsertime Videos

We have some scripts that can produce side-by-side comparison videos for you of the worst pairing of videos. You can find the script here: https://github.com/gmierz/moz-current-tests#browsertime-side-by-side-video-comparisons

Once the side-by-side comparison is produced, the video on the left is the old/base video, and the video on the right is the new video.

Bugs

Full Query

ID	Summary	Priority	Status
1543247	Add a `mach browsertime` command to make it easy to install and invoke browsertime	--	RESOLVED
1561939	[meta] MVP for browsertime on Windows 10 in CI	--	RESOLVED
1562870	Generate Conditioned profiles	P2	RESOLVED
1563074	Lift Android device performance tuning out of `testing/raptor/raptor.py`	P2	RESOLVED
1563090	Add a visualmetrics task type for computing visualmetrics from captured video	--	RESOLVED
1564256	Expose browsertime to generic-worker based test tasks	--	RESOLVED
1564282	Refactor raptor/raptor.py in preparation for browsertime	--	RESOLVED
1565316	Process browsertime results and dump out for perfherder	P1	RESOLVED
1566166	Add automation to install browsertime on Win 10 production performance hardware	P1	RESOLVED
1566171	Expose browsertime-* paths to Raptor test harness	P1	RESOLVED
1566174	Taskcluster config for raptor browsertime win10 page-load test	P2	VERIFIED
1568048	Lift mozproxy management out of `Raptor` and into `Perftest`	P1	RESOLVED
1572804	Begin to build browsertime support into Raptor framework	P1	RESOLVED
1574182	[meta] Browsertime support on android	P2	RESOLVED
1577905	Support warmload tests under --browsertime	P2	RESOLVED
1581580	Add limited set of --browsertime Raptor tests to make it easier to smoke-test changes	P1	RESOLVED
1583284	[meta] Validate --browsertime Raptor tasks against vanilla Raptor harness tasks	P1	RESOLVED
1584245	Target Chrome/Chromium browser under --browsertime	P2	RESOLVED
1585013	Add gecko profiling support to raptor-browsertime (desktop)	P1	RESOLVED
1585046	[meta] Expand the raptor-browsertime unit test coverage	P2	RESOLVED
1587500	Make raptor/browsertime return code setting more flexible	P3	NEW
1587761	Surface browsertime options in a generic way	P3	NEW
1587828	[meta] Refactor raptor-browsertime test INIs and taskcluster configs	P2	RESOLVED
1591560	Intermittent Browsertime warm page loads failing on raptor-tp6-facebook	P3	RESOLVED
1591875	Break Browsertime's `run_test` method into multiple specific methods	P2	RESOLVED
1592333	Improve raptor-browsertime page-load failure messages	P1	RESOLVED
1592633	Handling differences in prefs between browsertime and raptor-browsertime	P3	NEW
1592746	Raptor-browsertime perfherder_data includes all known measurements	P2	RESOLVED
1593109	Add chrome browsertime warm pageload tests	P3	RESOLVED
1599424	Convert vismet task output to a perfherder artifact	P1	RESOLVED
1601045	Intermittent raptor-main Critical: TEST-UNEXPECTED-FAIL: no raptor test results were found for raptor-tp6-amazon-firefox, raptor-tp6-facebook-firefox, raptor-tp6-google-firefox, raptor-tp6-youtube-firefox	P3	RESOLVED
1601414	Add application name and version in vismet perfherder results	P1	RESOLVED
1602577	Update in-tree Browsertime version	--	RESOLVED
1602657	Set appropriate value for post-startup-delay in (and only in) CI, for conditioned-profile runs	P1	RESOLVED
1602893	Retriggered browsertime vismet tasks only report results from the first task	P2	RESOLVED
1603152	For cold page-load set browser cycles to 3 when gecko profiling is enabled	P1	RESOLVED
1604638	don't use conditioned profiles in tests	P2	RESOLVED
1607851	Upgrade browsertime version used in production to the sitespeedio v8.0.0 tag	P1	RESOLVED
1608033	Get rid of custom geckodriver	P4	RESOLVED
1608863	Switch to sitespeed.io browsertime	P2	RESOLVED
1610389	Investigate Desktop Browsertime issues.	P3	RESOLVED
1612042	Enable fission for browsertime pageload tests.	P3	RESOLVED
1615347	Change location where temporary results are written to in Browsertime	P3	RESOLVED
1616845	Near-perma "OSError: symbolic link privilege not held" in fetch-content	P2	RESOLVED
1617239	Disable tier 3 browsertime tests on windows10-64-ref-hw-2017	P1	RESOLVED
1633336	Browsertime visual metrics tests with webrender enabled should run in the general_perf_testing cron	P3	RESOLVED
1633519	Raise browsertime error when it fails instead of generic ones	P1	RESOLVED
1645187	[meta] Make browsertime run faster	--	RESOLVED
1650133	[meta] Migrate webext desktop pageload tests to browsertime	P2	RESOLVED
1650871	Add all desktop pageload tests to browsertime	P1	RESOLVED
1664551	[meta] Speed up browsertime	P2	RESOLVED
1665911	Node crashes on Windows 7 browsertime tests	P2	RESOLVED
1665925	Tumblr pageload tests are perma-failing on browsertime	P2	RESOLVED
1668049	Prefix visual-metrics.py output with more information for treeherder	P2	RESOLVED
1684946	browsertime vismet tasks with --rebuild flag all reuse a single test result. Perfherder says Confidence is Infinity (high).	P3	REOPENED
1693108	Tumblr is permafailing in browsertime on windows2017 reference hardware	P2	RESOLVED
1710452	Migrate youtube-playback tests to browsertime	P1	RESOLVED
1732433	[browsertime] Remove scenario and power tests from webextension	P2	RESOLVED
1732634	Remove webext unittests manifest and associated code	P1	RESOLVED
1763457	Add support for profiling full run in CI	P3	NEW

60 Total; 5 Open (8.33%); 54 Resolved (90%); 1 Verified (1.67%);

@@ Line 1: / Line 1: @@
-Browsertime is a harness for running performance tests, similar to
+#REDIRECT [[TestEngineering/Performance/Raptor#WebExtension]]
-Mozilla's Raptor testing framework.  Browsertime is written in Node.js
-and uses Selenium WebDriver to drive multiple browsers including
+Browsertime is a harness for running performance tests, similar to Mozilla's Raptor testing framework. Browsertime is written in Node.js and uses Selenium WebDriver to drive multiple browsers including Chrome, Chrome for Android, Firefox, and Firefox for Android and GeckoView-based vehicles.
-Chrome, Chrome for Android, Firefox, and (pending the resolution of
-[https://bugzilla.mozilla.org/show_bug.cgi?id=1525126 Bug 1525126]
-and similar tickets) Firefox for Android and GeckoView-based vehicles.
 Source code:
-* Our current (vendored) Browsertime uses the [https://github.com/mozilla/browsertime Mozilla fork] of the [https://github.com/sitespeedio/browsertime canonical repo]
+* Our current Browsertime version uses the [https://github.com/sitespeedio/browsertime canonical repo].
 * In-tree: https://searchfox.org/mozilla-central/source/tools/browsertime and https://searchfox.org/mozilla-central/source/taskcluster/scripts/misc/browsertime.sh
-=== Page-load tests ===
 === Running Locally ===
 ==== Prerequisites ====
 * A local mozilla repository clone with a [https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions successful Firefox build] completed
+=== Setup ===
+'''Note that if you are running Raptor-Browsertime then it will get installed automatically and also updates itself.'''
 * Run <code>./mach browsertime --setup</code>
-* To check your setup, run <code>./mach browsertime --check</code>, which will output something like:<br>
-<code>
+* To check your setup, run <code>./mach browsertime --check</code>, which will output something like:
-ffmpeg:   OK<br>
-convert:  OK<br>
+  ffmpeg:   OK
-compare:  OK<br>
+  convert:  OK
-Pillow:   OK<br>
+  compare:  OK
-SSIM:     OK<br>
+  Pillow:   OK
-</code>
+  SSIM:     OK
-⚠️ If ffmpeg is listed as FAIL, you might want to try this:
-* <code>cd ~/.mozbuild/browsertime/ffmpeg-4.1.1-macos64-static/bin</code>
+* If <code>ffmpeg</code> is listed as FAIL, you might want to try this:
-* <code>chmod +x ffmpeg ffplay ffprobe</code>
+  cd ~/.mozbuild/browsertime/ffmpeg-4.1.1-macos64-static/bin
+  chmod +x ffmpeg ffplay ffprobe}}
 Now, try re-running <code>./mach browsertime --check</code>, with <code>ffmpeg</code> hopefully fixed
-Bugs which might further help are [https://bugzilla.mozilla.org/show_bug.cgi?id=1559168 bug 1559168], [https://bugzilla.mozilla.org/show_bug.cgi?id=1559727 bug 1559727], and [https://bugzilla.mozilla.org/show_bug.cgi?id=1574964 bug 1574964], for starters
+* For other issues, see if <code>./mach browsertime --setup --clobber</code> fixes it, or deleting the <tt>~/.mozbuild/browsertime</tt> folder and running the aforementioned command.
+* If you aren't running visual metrics, then failures in <code>Pillow</code> and <code>SSIM</code> can be ignored.
+If <code>convert</code> and <code>compare</code> are also <code>FAIL</code> bugs which might further help are {{bug|1559168}}, {{bug|1559727}}, and {{bug|1574964}}, for starters. If none of the bugs are related to the issue, please file a bug <tt>Testing :: Raptor</tt>.
 * If you plan on running Browsertime on Android, your Android device must already be set up (see more below in the Android section)
 ==== Running on Firefox Desktop ====
+=== Page-load tests ===
+There are two ways to run performance tests through browsertime listed below. '''Note that `./mach browsertime` should not be used when debugging performance issues with profiles as it does not do symbolication.'''
+* Raptor-Browsertime (recommended):<br>
+ ./mach raptor --browsertime -t google-search
 * Browsertime-"native":<br>
    ./mach browsertime https://www.sitespeed.io --firefox.binaryPath '/Users/{userdir}/moz_src/mozilla-unified/obj-x86_64-apple-darwin18.7.0/dist/Nightly.app/Contents/MacOS/firefox'
+=== Benchmark tests ===
 * Raptor-wrapped:<br>
-  ./mach raptor --browsertime -t raptor-tp6-google-firefox
+  ./mach raptor -t raptor-speedometer --browsertime
-==== Running on Fennec ====
+==== Running on Android ====
-   .
+Running on Raptor-Browsertime (recommended):
-==== Running on Fenix/Firefox Preview ====
+* Running on Fenix
+   ./mach raptor --browsertime -t amazon --app fenix --binary org.mozilla.fenix
+* Running on Geckoview
+  ./mach raptor --browsertime -t amazon --app geckoview --binary org.mozilla.geckoview_example
+Running on vanilla Browsertime:
+* Running on Fenix/Firefox Preview
    ./mach browsertime --android --browser firefox --firefox.android.package org.mozilla.fenix.debug --firefox.android.activity org.mozilla.fenix.IntentReceiverActivity https://www.sitespeed.io
-==== Running on the GeckoView Example app ====
+* Running on the GeckoView Example app
    ./mach browsertime --android --browser firefox https://www.sitespeed.io
-==== Running on Reference Browser ====
-  .
 ==== Running on Google Chrome ====
+Chrome releases are tied to a specific version of ChromeDriver -- you will need to ensure the two are aligned.
+There are two ways of doing this:
+. Download the ChromeDriver that matches the chrome you wish to run from https://chromedriver.chromium.org/ and specify the path:
+  ./mach browsertime https://www.sitespeed.io -b chrome --chrome.chromedriverPath <PATH/TO/VERSIONED/CHROMEDRIVER>
+. Upgrade the ChromeDriver version in <code>tools/browsertime/package-lock.json </code> (see https://www.npmjs.com/package/@sitespeed.io/chromedriver for versions).
+Run <code>npm install</code>.
+Launch vanilla Browsertime as follows:
    ./mach browsertime https://www.sitespeed.io -b chrome
+Or for Raptor-Browsertime (use <code>chrome</code> for desktop, and <code>chrome-m</code> for mobile):
+  ./mach raptor --browsertime -t amazon --app chrome --browsertime-chromedriver <PATH/TO/CHROMEDRIVER>
 === More Examples ===
 [https://github.com/mozilla/browsertime/tree/master/docs/examples Browsertime docs]
 === Running Browsertime on Try ===
-Include the <tt>--browsertime</tt> flag to <tt>mach try fuzzy</tt>.  As of [https://bugzilla.mozilla.org/show_bug.cgi?id=1566174 bug 1566174] this will arrange for the jobs to include dependencies for Browsertime (Node.js, Browsertime <tt>node_modules</tt>, <tt>geckodriver</tt>, <tt>chromedriver</tt>, etc) and for the Raptor harness to run the given pageload test using Browsertime.
+You can run all of our browsertime pageload tests through <code>./mach try fuzzy --full</code>. We use chimera mode in these tests which means that both cold and warm pageload variants are running at the same time.
+For example:
+  ./mach try fuzzy -q "'g5 'imdb 'geckoview 'vismet '-wr 'shippable"
+=== Retriggering Browsertime Visual Metrics Tasks ===
+You can retrigger Browsertime tasks just like you retrigger any other tasks from Treeherder (using the retrigger buttons, add-new-jobs, retrigger-multiple, etc.).
+When you retrigger the Browsertime test task, it will trigger a new vismet task as well. If you retrigger a Browsertime vismet task, then it will cause the test task to be retriggered and a new vismet task will be produced from there. This means that both of these tasks are treated as "one" task when it comes to retriggering them.
+There is only one path that still doesn't work for retriggering Browsertime tests and that's happens when you use `--rebuild X` in a try push submission.
+For details on how we previously retriggered visual metrics tasks see [[/VisualMetrics/]] (this will stay here for a few months just in case).
+=== Gecko Profiling with Browsertime ===
+To run gecko profiling using Raptor-Browsertime you can add the <code>--gecko-profile</code> flag to any command and you will get profiles from the test (with the profiler page opening in the browser automatically). This method also performs symbolication for you. For example:
+  ./mach raptor --browsertime -t amazon --gecko-profile
+Note that vanilla Browsertime does support Gecko Profiling but '''it does not symbolicate the profiles''' so it is '''not recommended''' to use for debugging performance regressions/improvements.
+=== Upgrading Browsertime In-Tree ===
+To upgrade the browsertime version used in-tree you can run, then commit the changes made to <code>package.json</code> and <code>package-lock.json</code>:
+  ./mach browsertime --update-upstream-url <TARBALL-URL>
+Here is a sample URL that we can update to: https://github.com/sitespeedio/browsertime/tarball/89771a1d6be54114db190427dbc281582cba3d47
+To test the upgrade, run a raptor test locally (with and without visual-metrics <code>--browsertime-visualmetrics</code> if possible) and test it on try with at least one test on desktop and mobile.
-'''Not all Raptor tasks use Browsertime meaningfully.  Right now, only cold pageload tests are supported.'''  See [https://bugzilla.mozilla.org/show_bug.cgi?id=1577905 bug 1577905] for warm pageload tests, for example.
+=== Finding the Geckodriver Being Used ===
+If you're looking for the latest geckodriver being used there are two ways:
+* Find the latest one from here: https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=geckodriver
+* Alternatively, if you're trying to figure out which geckodriver a given CI task is using, you can click on the browsertime task in treeherder, and then click on the `Task` id in the bottom left of the pop-up interface. Then in the window that opens up, click on `See more` in the task details tab on the left, this will show you the dependent tasks with the latest toolchain-geckodriver being used. There's an Artifacts drop down on the right hand side for the toolchain-geckodriver task that you can find the latest geckodriver in.
-For example:
+If you're trying to test Browsertime with a new geckodriver, you can do either of the following:
-  ./mach try fuzzy --browsertime -q "'test-android-hw-g5-7-0-arm7-api-16/opt-raptor-tp6m-1-geckoview-e10s"
+* Request a new geckodriver build in your try run (i.e. through <code>./mach try fuzzy</code>).
+* Trigger a new geckodriver in a try push, then trigger the browsertime tests which will then use the newly built version in the try push.
+=== Comparing Before/After Browsertime Videos ===
+We have some scripts that can produce side-by-side comparison videos for you of the worst pairing of videos. You can find the script here: https://github.com/gmierz/moz-current-tests#browsertime-side-by-side-video-comparisons
+Once the side-by-side comparison is produced, the video on the left is the old/base video, and the video on the right is the new video.
 === Bugs ===