TestEngineering/Performance/Raptor/Browsertime

Browsertime is a harness for running performance tests, similar to Mozilla's Raptor testing framework. Browsertime is written in Node.js and uses Selenium WebDriver to drive multiple browsers including Chrome, Chrome for Android, Firefox, and (pending the resolution of Bug 1525126 and similar tickets) Firefox for Android and GeckoView-based vehicles.

Source code:

Our current (vendored) Browsertime uses the Mozilla fork of the canonical repo
In-tree: https://searchfox.org/mozilla-central/source/tools/browsertime and https://searchfox.org/mozilla-central/source/taskcluster/scripts/misc/browsertime.sh

Page-load tests

Running Locally

Prerequisites

A local mozilla repository clone with a successful Firefox build completed
Run ./mach browsertime --setup

⚠️ If ffmpeg is listed as FAIL, you might want to try this:

cd ~/.mozbuild
cd browsertime
cd bin
chmod +x ffmpeg ffplay ffprobe

Bugs which might further help are bug 1559168, bug 1559727, and bug 1574964, for starters

If you plan on running Browsertime on Android, your Android device must already be set up (see more below in the Android section)

Running on Firefox Desktop

sdonner$ ./mach browsertime https://www.sitespeed.io --firefox.binaryPath '/Users/sdonner/moz_src/mozilla-unified/obj-x86_64-apple-darwin18.7.0/dist/Nightly.app/Contents/MacOS/firefox'

Running on Google Chrome

./mach browsertime https://www.sitespeed.io -b chrome

Running on Fenix/Firefox Preview

./mach browsertime --android --browser firefox --firefox.android.package org.mozilla.fenix.debug --firefox.android.activity org.mozilla.fenix.IntentReceiverActivity https://www.sitespeed.io

Running on the GeckoView Example app

./mach browsertime --android --browser firefox https://www.sitespeed.io

More Examples

Browsertime docs

Running Browsertime on Try

Pending fix in bug 1566174

Bugs

Full Query

ID	Summary	Priority	Status
1543247	Add a `mach browsertime` command to make it easy to install and invoke browsertime	--	RESOLVED
1561939	[meta] MVP for browsertime on Windows 10 in CI	--	RESOLVED
1562870	Generate Conditioned profiles	P2	RESOLVED
1563074	Lift Android device performance tuning out of `testing/raptor/raptor.py`	P2	RESOLVED
1563090	Add a visualmetrics task type for computing visualmetrics from captured video	--	RESOLVED
1564256	Expose browsertime to generic-worker based test tasks	--	RESOLVED
1564282	Refactor raptor/raptor.py in preparation for browsertime	--	RESOLVED
1565316	Process browsertime results and dump out for perfherder	P1	RESOLVED
1566166	Add automation to install browsertime on Win 10 production performance hardware	P1	RESOLVED
1566171	Expose browsertime-* paths to Raptor test harness	P1	RESOLVED
1566174	Taskcluster config for raptor browsertime win10 page-load test	P2	VERIFIED
1568048	Lift mozproxy management out of `Raptor` and into `Perftest`	P1	RESOLVED
1572804	Begin to build browsertime support into Raptor framework	P1	RESOLVED
1574182	[meta] Browsertime support on android	P2	RESOLVED
1577905	Support warmload tests under --browsertime	P2	RESOLVED
1581580	Add limited set of --browsertime Raptor tests to make it easier to smoke-test changes	P1	RESOLVED
1583284	[meta] Validate --browsertime Raptor tasks against vanilla Raptor harness tasks	P1	RESOLVED
1584245	Target Chrome/Chromium browser under --browsertime	P2	RESOLVED
1585013	Add gecko profiling support to raptor-browsertime (desktop)	P1	RESOLVED
1585046	[meta] Expand the raptor-browsertime unit test coverage	P2	RESOLVED
1587500	Make raptor/browsertime return code setting more flexible	P3	NEW
1587761	Surface browsertime options in a generic way	P3	NEW
1587828	[meta] Refactor raptor-browsertime test INIs and taskcluster configs	P2	RESOLVED
1591560	Intermittent Browsertime warm page loads failing on raptor-tp6-facebook	P3	RESOLVED
1591875	Break Browsertime's `run_test` method into multiple specific methods	P2	RESOLVED
1592333	Improve raptor-browsertime page-load failure messages	P1	RESOLVED
1592633	Handling differences in prefs between browsertime and raptor-browsertime	P3	NEW
1592746	Raptor-browsertime perfherder_data includes all known measurements	P2	RESOLVED
1593109	Add chrome browsertime warm pageload tests	P3	RESOLVED
1599424	Convert vismet task output to a perfherder artifact	P1	RESOLVED
1601045	Intermittent raptor-main Critical: TEST-UNEXPECTED-FAIL: no raptor test results were found for raptor-tp6-amazon-firefox, raptor-tp6-facebook-firefox, raptor-tp6-google-firefox, raptor-tp6-youtube-firefox	P3	RESOLVED
1601414	Add application name and version in vismet perfherder results	P1	RESOLVED
1602577	Update in-tree Browsertime version	--	RESOLVED
1602657	Set appropriate value for post-startup-delay in (and only in) CI, for conditioned-profile runs	P1	RESOLVED
1602893	Retriggered browsertime vismet tasks only report results from the first task	P2	RESOLVED
1603152	For cold page-load set browser cycles to 3 when gecko profiling is enabled	P1	RESOLVED
1604638	don't use conditioned profiles in tests	P2	RESOLVED
1607851	Upgrade browsertime version used in production to the sitespeedio v8.0.0 tag	P1	RESOLVED
1608033	Get rid of custom geckodriver	P4	RESOLVED
1608863	Switch to sitespeed.io browsertime	P2	RESOLVED
1610389	Investigate Desktop Browsertime issues.	P3	RESOLVED
1612042	Enable fission for browsertime pageload tests.	P3	RESOLVED
1615347	Change location where temporary results are written to in Browsertime	P3	RESOLVED
1616845	Near-perma "OSError: symbolic link privilege not held" in fetch-content	P2	RESOLVED
1617239	Disable tier 3 browsertime tests on windows10-64-ref-hw-2017	P1	RESOLVED
1633336	Browsertime visual metrics tests with webrender enabled should run in the general_perf_testing cron	P3	RESOLVED
1633519	Raise browsertime error when it fails instead of generic ones	P1	RESOLVED
1645187	[meta] Make browsertime run faster	--	RESOLVED
1650133	[meta] Migrate webext desktop pageload tests to browsertime	P2	RESOLVED
1650871	Add all desktop pageload tests to browsertime	P1	RESOLVED
1664551	[meta] Speed up browsertime	P2	RESOLVED
1665911	Node crashes on Windows 7 browsertime tests	P2	RESOLVED
1665925	Tumblr pageload tests are perma-failing on browsertime	P2	RESOLVED
1668049	Prefix visual-metrics.py output with more information for treeherder	P2	RESOLVED
1684946	browsertime vismet tasks with --rebuild flag all reuse a single test result. Perfherder says Confidence is Infinity (high).	P3	REOPENED
1693108	Tumblr is permafailing in browsertime on windows2017 reference hardware	P2	RESOLVED
1710452	Migrate youtube-playback tests to browsertime	P1	RESOLVED
1732433	[browsertime] Remove scenario and power tests from webextension	P2	RESOLVED
1732634	Remove webext unittests manifest and associated code	P1	RESOLVED
1763457	Add support for profiling full run in CI	P3	NEW

60 Total; 5 Open (8.33%); 54 Resolved (90%); 1 Verified (1.67%);