TestEngineering/Performance/Raptor
Raptor
Raptor is a new performance testing framework for running browser pageload and browser benchmark tests. The core of Raptor was designed as a browser extension, therefore Raptor is cross-browser compatible and is currently running in production (tier 2) on Firefox and Google Chrome.
Raptor supports two types of performance tests: pageload tests, and standard benchmark tests. Pageload tests basically involve loading a specific web page and measuring the load performance (i.e. time-to-first-non-blank-paint). Standard benchmarks are third-party tests (i.e. Speedometer) that we have integrated into Raptor to run per-commit in our production CI.
For pageload tests, instead of using live web pages for performance testing, Raptor uses a tool called [Mitmproxy]. Mitmproxy allows us to record and playback test pages via a local Firefox proxy. The Mitmproxy recordings are stored on tooltool and are automatically downloaded by Raptor when they are required for a test.
Running Locally: Prerequisites
In order to run Raptor on a local machine you need:
- A local mozilla repository clone with a successful Firefox build completed
- GIT needs to be in the path in the terminal that you build Firefox / run Raptor from, as Raptor uses GIT to check out a local copy of some of the source for some of the performance benchmarks
- If you plan on running Raptor tests on Google Chrome, you need a local install of Google Chrome and know the path to the chrome binary
- If you plan on running Raptor on android, your android device must already be setup (see more below in the Android section)
Running Locally on Firefox
To run Raptor locally just build Firefox and then run:
mozilla-central$ ./mach raptor-test --test <raptor-test-name>
For example to run the raptor tp6 pageload test locally just use:
mozilla-central$ ./mach raptor-test --test raptor-tp6
Raptor test results will be found locally in <your-repo>/testing/mozharness/build/raptor.json.
Running Locally on Google Chrome
To run Raptor locally on Google Chrome, make sure you already have a local version of Google Chrome installed, and then from within your mozilla-repo run:
mozilla-central$ ./mach raptor-test --test <raptor-test-name> --app=chrome --binary="<path to google chrome binary>"
For example to run the raptor-speedometer benchmark on Google Chrome use:
mozilla-central$ ./mach raptor-test --test raptor-speedometer --app=chrome --binary="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome
Raptor test results will be found locally in <your-repo>/testing/mozharness/build/raptor.json.
Running Locally on Android Geckoview
When running Raptor tests on a local android device, Raptor is expecting the device to already be setup and ready to go.
First ensure your local host machine has the Android SDK/Tools (i.e. ADB) installed. Check if it is already installed by attaching your android device to USB and running:
adb devices
If your device serial number is listed then you're set. If ADB is not found you can install it by running (in your local mozilla development repo):
./mach bootstrap
Then in bootstrap select the option for "Firefox for Android Artifact Mode" and that will install the required tools (no need to do an actual build).
Next make sure your android device is ready to go. Local android device pre-requisites are:
- Device is rooted
- Device is in 'superuser' mode
- The geckoview example app is already installed on the device. Download the geckoview_example.apk from the appropriate android build on treeherder, then install it on your device i.e.:
adb install -g ../Downloads/geckoview_example.apk
The '-g' flag will automatically set all application permissions ON which is required. Note, when updating the geckoview example app, you must uninstall the existing one first, i.e.:
adb uninstall org.mozilla.geckoview_example
Once your android device is ready, and attached to local USB, from within your local mozilla repo use the following command line to run speedometer:
./mach raptor-test --test raptor-speedometer --app=geckoview --binary="org.mozilla.geckoview_example"
Note: Speedometer on android geckoview is currently running on two devices in production - the Google Pixel 2 and the Moto G5 - therefore it is not guaranteed that it will run successfully on all/other untested android devices. There is an intermittent failure on the Moto G5 where speedometer just stalls (Bug 1492222).
A couple of notes about debugging:
- Raptor browser extension console messages do appear in adb logcat via the GeckoConsole - so this is handy:
adb logcat | grep GeckoConsole
- You can also debug Raptor on android using the Firefox WebIDE, click on the android device listed under "USB Devices" and then "Main Process" or the 'localhost: Speedometer.." tab process
Raptor test results will be found locally in <your-repo>/testing/mozharness/build/raptor.json.
Running Raptor on Try
Raptor tests can be run on try on both Firefox and Google Chrome. (Raptor pageload-type tests are not supported on Google Chrome yet, as mentioned above).
Note: Raptor is currently 'tier 2' on Treeherder, which means to see the Raptor test jobs you need to ensure 'tier 2' is selected / turned on in the Treeherder 'Tiers' menu.
The easiest way to run Raptor tests on try is to use mach try fuzzy:
mozilla-central$ $ ./mach try fuzzy
Then type 'raptor' and select which Raptor tests (and on what platforms) you wish to run.
To see the Raptor test results on your try run:
- In treeherder select one of the Raptor test jobs (i.e. 'sp' in 'Rap-e10s', or 'Rap-C-e10s')
- Below the jobs, click on the "Performance" tab; you'll see the aggregated results listed
- If you wish to see the raw replicates, click on the "Job Details" tab, and select the "perfherder-data.json" artifact
Raptor Desktop Tests
Currently the following Raptor tests are available to run on the desktop browser(s).
raptor-gdocs
- contact: :rwood, :jmaher
- type: pageload
- browsers: Firefox only (for now)
- measuring: time-to-first-non-blank-paint, and hero element load time
- reporting: For each test page both fnbpaint and time to hero element display are measured. Each pagecycle reports the mean of those two values (in MS). There are 25 pagecycles, with the first pagecycle being dropped due to the initial extra loading time/noise. The overall result reported for each test page is the median of the 24 values reported for each pagecycle (in MS).
The raptor-gdocs test loads three pages (a google document, a google sheets spreadsheet, and a google shares presentation) 25x each and measures their load performance (fnbpaint, and hero element). To test all of the pages, run 'raptor-gdocs'. To only test one of the individual pages, run the corresponding raptor test ('raptor-google-docs-firefox', 'raptor-google-sheets-firefox', or 'raptor-google-slides-firefox').
In each of the test page (doc/slides/sheets) recordings, there is a [hero element] located as part of the blue 'Sign In' button on the top right.
The pages are played back from [Mitmproxy] recordings. If you need the HTML page source (outside of the Mitmproxy recording) for debugging, the HTML can be found in our perf-automation github repo.
To see the specific google document URLs and other raptor test settings, you can view the raptor-gdocs.ini.
raptor-motionmark
- contact: ?
- type: benchmark
- browsers: Firefox and Google Chrome
- measuring: benchmark measuring the time to animate complex scenes
- summarization:
- subtest: FPS from the subtest, each subtest is run for 15 seconds, repeat this 5 times and report the median value
- suite: we take a geometric mean of all the subtests (9 for animometer, 11 for html suite)
Note: There are two tests for motionmark, the test names for the Raptor command line are 'raptor-motionmark-animometer and 'raptor-motionmark-htmlsuite'.
raptor-speedometer
- contact: :selena
- type: benchmark
- browsers: Firefox and Google Chrome
- measuring: responsiveness of web applications
- reporting: runs/minute score
- data: there are 16 subtests in Speedometer; each of these are made up of 9 internal benchmarks.
- summarization:
- subtest: For all of the 16 subtests, we collect the sum of all their internal benchmark results.
- score: geometric mean of the 16 sums
This is the Speedometer javascript benchmark taken verbatim and slightly modified to fit into our pageloader extension and raptor harness.
raptor-stylebench
- contact: :emilio
- type: benchmark
- browsers: Firefox and Google Chrome
- measuring: speed of dynamic style recalculation
- reporting: runs/minute score
raptor-sunspider
- contact: ?
- type: benchmark
- browsers: Firefox and Google Chrome
- TODO
raptor-tp6
- contact: :rwood, :jmaher
- type: pageload
- browsers: Firefox only (for now)
- measuring: time-to-first-non-blank-paint, and hero element load time
- reporting: For each test page both fnbpaint and time to hero element display are measured. Each pagecycle reports the mean of those two values (in MS). There are 25 pagecycles, with the first pagecycle being dropped due to the initial extra loading time/noise. The overall result reported for each test page is the median of the 24 values reported for each pagecycle (in MS).
The raptor-tp6 test loads four pages (amazon, facebook, google, youtube) 25x each and measures their load performance (fnbpaint and hero element display). To test all of the pages, run 'raptor-tp6'. To only test one of the individual pages, run the corresponding raptor test ('raptor-tp6-amazon-firefox', 'raptor-tp6-google-firefox', 'raptor-tp6-facebook-firefox', or 'raptor-tp6-youtube-firefox').
The pages are played back from [Mitmproxy] recordings. If you need the HTML page source (outside of the Mitmproxy recording) for debugging, the HTML can be found in our perf-automation github repo.
The page recordings contain [hero elements], located a follows:
raptor-tp6-amazon: string description element for first laptop in search results
raptor-tp6-facebook: on the Facebook 'Home' icon
raptor-tp6-google: bigger photo of Obama in search results towards top right
raptor-tp6-youtube: YouTube logo on the top left
To see the specific test URLs and other raptor test settings, you can view the raptor-tp6.ini.
raptor-webaudio
- contact: ?
- type: benchmark
- browsers: Firefox and Google Chrome
- TODO
Debugging the Raptor Web Extension
When developing on Raptor and debugging, there's often a need to look at the output coming from the Raptor Web Extension. Here are some pointers to help.
Debugging when Running on Firefox Desktop
The main Raptor runner is 'runner.js' which is inside the web extension. The code that actually captures the performance measures is in the web extension content code 'measure.js'.
In order to retrieve the console.log() output from the Raptor runner, do the following:
- Invoke Raptor locally via ./mach raptor-test
- During the 30 second Raptor pause which happens right after Firefox has started up, in the ALREADY OPEN current tab, type "about:debugging" for the URL.
- On the debugging page that appears, make sure "Add-ons" is selected on the left (default).
- Turn ON the "Enable add-on debugging" check-box
- Then scroll down the page until you see the Raptor web extension in the list of currently-loaded add-ons. Under "Raptor" click the blue "Debug" link.
- A new window will open in a minute, and click the "console" tab
To retrieve the console.log() output from the Raptor content 'measure.js' code:
- As soon as Raptor opens the new test tab (and the test starts running / or the page starts loading), in Firefox just choose "Tools => Web Developer => Web Console", and select the "console' tab.
Raptor automatically closes the test tab and the entire browser after test completion; which will close any open debug consoles. In order to have more time to review the console logs, Raptor can be temporarily hacked locally in order to prevent the test tab and browser from being closed. Currently this must be done manually, as follows:
- In the Raptor web extension runner, comment out the line that closes the test tab in the test clean-up. That line of code is here.
- Add a return statement at the top of the Raptor control server method that shuts-down the browser, the browser shut-down method is here.
For benchmark type tests (i.e. speedometer, motionmark, etc.) Raptor doesn't inject 'measure.js' into the test page content; instead it injects 'benchmark-relay.js' into the benchmark test content. Benchmark-relay is as it sounds; it basically relays the test results coming from the benchmark test, to the Raptor web extension runner. Viewing the console.log() output from benchmark-relay is done the same was as noted for the 'measure.js' content above.
Note, Bug 1470450 is on file to add a debug mode to Raptor that will automatically grab the web extension console output and dump it to the terminal (if possible) that will make debugging much easier.
Debugging TP6 and Killing the Mitmproxy Server
Regarding debugging Raptor pageload tests that use Mitmproxy (i.e. tp6, gdocs). If Raptor doesn't finish naturally and doesn't stop the Mitmproxy tool, the next time you attempt to run Raptor it might fail out with this error:
INFO - Error starting proxy server: OSError(48, 'Address already in use')
INFO - raptor-mitmproxy Aborting: mitmproxy playback process failed to start, poll returned: 1
That just means the Mitmproxy server was already running before so it couldn't startup. In this case, you need to kill the Mitmproxy server processes, i.e:
mozilla-unified rwood$ ps -ax | grep mitm
5439 ttys000 0:00.09 /Users/rwood/mozilla-unified/obj-x86_64-apple-darwin17.7.0/testing/raptor/mitmdump -k -q -s /Users/rwood/mozilla-unified/testing/raptor/raptor/playback/alternate-server-replay.py /Users/rwood/mozilla-unified/obj-x86_64-apple-darwin17.7.0/testing/raptor/amazon.mp
5440 ttys000 0:01.64 /Users/rwood/mozilla-unified/obj-x86_64-apple-darwin17.7.0/testing/raptor/mitmdump -k -q -s /Users/rwood/mozilla-unified/testing/raptor/raptor/playback/alternate-server-replay.py /Users/rwood/mozilla-unified/obj-x86_64-apple-darwin17.7.0/testing/raptor/amazon.mp
5509 ttys000 0:00.01 grep mitm
Then just kill the first mitm process in the list and that's sufficient:
mozilla-unified rwood$ kill 5439
Now when you run Raptor again, the Mitmproxy server will be able to start.
Debugging when Running on Firefox for Android
Be sure to read the above section first on how to debug the Raptor web extension when running on Firefox Desktop.
When running Raptor tests on Firefox on Android (i.e. geckoview), to see the console.log() output from the Raptor web extension, do the following:
- With your android device (i.e. Google Pixel 2) all setup and connected to USB, invoke the Raptor test normally via ./mach raptor-test
- Startup a local copy of the Firefox Nightly Desktop browser
- In Firefox Desktop choose "Tools => Web Developer => WebIDE"
- In the Firefox WebIDE dialog that appears, look under "USB Devices" listed on the top right. If your device is not there, there may be a link to install remote device tools - if that link appears click it and let that install.
- Under "USB Devices" on the top right your android device should be listed (i.e. "Firefox Custom on Android Pixel 2" - click on your device.
- The debugger opens. On the left side click on "Main Process", and click the "console" tab below - and the Raptor runner output will be included there.
- On the left side under "Tabs" you'll also see an option for the active tab/page, select that and the Raptor content console.log() output should be included there.
Also note: When debugging Raptor on Android, the 'adb logcat' is very useful. More specifically for 'geckoview', the output (including for Raptor) is prefixed with "GeckoConsole" - so this command is very handy:
adb logcat | grep GeckoConsole