MozillaWiki

TestEngineering/Performance/Raptor: Difference between revisions

Page Discussion

TestEngineering/Performance/Raptor (view source)

Revision as of 11:07, 26 July 2021

1,715 bytes removed ,  26 July 2021
file page for deletion, content has been migrated to source docs
(Change all ./mach raptor-test calls to ./mach raptor and add a message about running locally at the top that also mentions the possibility of raptor-test being deprecated in the future.)
(file page for deletion, content has been migrated to source docs)
 
(35 intermediate revisions by 8 users not shown)
Line 1: Line 1:
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/raptor.html|date=2021-07-26}}
[[Image:Raptor.png|frameless|right]]
[[Image:Raptor.png|frameless|right]]


Raptor is a performance-testing framework for running browser pageload and browser benchmark tests. The core of Raptor was designed as a browser extension, therefore Raptor is cross-browser compatible and is currently running in production on Firefox Desktop, Firefox Android GeckoView, and on Google Chromium.
Raptor is a performance-testing framework for running browser pageload and browser benchmark tests. Raptor is cross-browser compatible and is currently running in production on Firefox Desktop, Firefox Android GeckoView, Fenix, Reference Browser, Chromium, and Chrome.


* Contact: Rob Wood [rwood]
* Contact: Dave Hunt [:davehunt]
* Source code: https://searchfox.org/mozilla-central/source/testing/raptor
* Source code: https://searchfox.org/mozilla-central/source/testing/raptor
* Good first bugs: https://codetribute.mozilla.org/projects/automation?project%3DRaptor
* Good first bugs: https://codetribute.mozilla.org/projects/automation?project%3DRaptor
Line 9: Line 11:
Raptor currently supports three test types: 1) page-load performance tests, 2) standard benchmark-performance tests, and 3) "scenario"-based tests, such as power, CPU, and memory-usage measurements on Android (and desktop?).
Raptor currently supports three test types: 1) page-load performance tests, 2) standard benchmark-performance tests, and 3) "scenario"-based tests, such as power, CPU, and memory-usage measurements on Android (and desktop?).


Locally, raptor can be invoked with either of the following commands - raptor-test may be deprecated in the future:
Locally, Raptor can be invoked with the following command:
 
     ./mach raptor
     ./mach raptor
    ./mach raptor-test


== Browsertime ==
Browsertime is a harness for running performance tests, similar to Mozilla's Raptor testing framework. Browsertime is written in Node.js and uses Selenium WebDriver to drive multiple browsers including Chrome, Chrome for Android, Firefox, and Firefox for Android and GeckoView-based vehicles.
Source code:
* Our current Browsertime version uses the [https://github.com/sitespeedio/browsertime canonical repo].
* In-tree: https://searchfox.org/mozilla-central/source/tools/browsertime and https://searchfox.org/mozilla-central/source/taskcluster/scripts/misc/browsertime.sh
=== Running Locally ===
==== Prerequisites ====
* A local mozilla repository clone with a [https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions successful Firefox build] completed
=== Setup ===
'''Note that if you are running Raptor-Browsertime then it will get installed automatically and also updates itself.'''
* Run <code>./mach browsertime --setup</code>
* To check your setup, run <code>./mach browsertime --check</code>, which will output something like:
  ffmpeg:  OK
  convert:  OK
  compare:  OK
  Pillow:  OK
  SSIM:    OK
* If <code>ffmpeg</code> is listed as FAIL, you might want to try this:
  cd ~/.mozbuild/browsertime/ffmpeg-4.1.1-macos64-static/bin
  chmod +x ffmpeg ffplay ffprobe}}
Now, try re-running <code>./mach browsertime --check</code>, with <code>ffmpeg</code> hopefully fixed
* For other issues, see if <code>./mach browsertime --setup --clobber</code> fixes it, or deleting the <tt>~/.mozbuild/browsertime</tt> folder and running the aforementioned command.
* If you aren't running visual metrics, then failures in <code>Pillow</code> and <code>SSIM</code> can be ignored.
If <code>convert</code> and <code>compare</code> are also <code>FAIL</code> bugs which might further help are {{bug|1559168}}, {{bug|1559727}}, and {{bug|1574964}}, for starters. If none of the bugs are related to the issue, please file a bug <tt>Testing :: Raptor</tt>.
* If you plan on running Browsertime on Android, your Android device must already be set up (see more below in the Android section)
==== Running on Firefox Desktop ====
=== Page-load tests ===
There are two ways to run performance tests through browsertime listed below. '''Note that `./mach browsertime` should not be used when debugging performance issues with profiles as it does not do symbolication.'''
* Raptor-Browsertime (recommended):<br>
./mach raptor --browsertime -t google-search
* Browsertime-"native":<br>
  ./mach browsertime https://www.sitespeed.io --firefox.binaryPath '/Users/{userdir}/moz_src/mozilla-unified/obj-x86_64-apple-darwin18.7.0/dist/Nightly.app/Contents/MacOS/firefox'
=== Benchmark tests ===
* Raptor-wrapped:<br>
./mach raptor -t raptor-speedometer --browsertime
==== Running on Android ====
Running on Raptor-Browsertime (recommended):
* Running on Fenix
  ./mach raptor --browsertime -t amazon --app fenix --binary org.mozilla.fenix
* Running on Geckoview
  ./mach raptor --browsertime -t amazon --app geckoview --binary org.mozilla.geckoview_example
Running on vanilla Browsertime:
* Running on Fenix/Firefox Preview
  ./mach browsertime --android --browser firefox --firefox.android.package org.mozilla.fenix.debug --firefox.android.activity org.mozilla.fenix.IntentReceiverActivity https://www.sitespeed.io
* Running on the GeckoView Example app
  ./mach browsertime --android --browser firefox https://www.sitespeed.io
==== Running on Google Chrome ====
Chrome releases are tied to a specific version of ChromeDriver -- you will need to ensure the two are aligned.
There are two ways of doing this:
1. Download the ChromeDriver that matches the chrome you wish to run from https://chromedriver.chromium.org/ and specify the path:
  ./mach browsertime https://www.sitespeed.io -b chrome --chrome.chromedriverPath <PATH/TO/VERSIONED/CHROMEDRIVER>
2. Upgrade the ChromeDriver version in <code>tools/browsertime/package-lock.json </code> (see https://www.npmjs.com/package/@sitespeed.io/chromedriver for versions).
Run <code>npm install</code>.
Launch vanilla Browsertime as follows:
  ./mach browsertime https://www.sitespeed.io -b chrome
Or for Raptor-Browsertime (use <code>chrome</code> for desktop, and <code>chrome-m</code> for mobile):
  ./mach raptor --browsertime -t amazon --app chrome --browsertime-chromedriver <PATH/TO/CHROMEDRIVER>
=== More Examples ===
[https://github.com/mozilla/browsertime/tree/master/docs/examples Browsertime docs]
=== Running Browsertime on Try ===
You can run all of our browsertime pageload tests through <code>./mach try fuzzy --full</code>. We use chimera mode in these tests which means that both cold and warm pageload variants are running at the same time.
For example:
  ./mach try fuzzy -q "'g5 'imdb 'geckoview 'vismet '-wr 'shippable"
=== Retriggering Browsertime Visual Metrics Tasks ===
You can retrigger Browsertime tasks just like you retrigger any other tasks from Treeherder (using the retrigger buttons, add-new-jobs, retrigger-multiple, etc.).
When you retrigger the Browsertime test task, it will trigger a new vismet task as well. If you retrigger a Browsertime vismet task, then it will cause the test task to be retriggered and a new vismet task will be produced from there. This means that both of these tasks are treated as "one" task when it comes to retriggering them.
There is only one path that still doesn't work for retriggering Browsertime tests and that's happens when you use `--rebuild X` in a try push submission.
For details on how we previously retriggered visual metrics tasks see [[/VisualMetrics/]] (this will stay here for a few months just in case).
=== Gecko Profiling with Browsertime ===
To run gecko profiling using Raptor-Browsertime you can add the <code>--gecko-profile</code> flag to any command and you will get profiles from the test (with the profiler page opening in the browser automatically). This method also performs symbolication for you. For example:
  ./mach raptor --browsertime -t amazon --gecko-profile
Note that vanilla Browsertime does support Gecko Profiling but '''it does not symbolicate the profiles''' so it is '''not recommended''' to use for debugging performance regressions/improvements.
=== Upgrading Browsertime In-Tree ===
To upgrade the browsertime version used in-tree you can run, then commit the changes made to <code>package.json</code> and <code>package-lock.json</code>:
  ./mach browsertime --update-upstream-url <TARBALL-URL>
Here is a sample URL that we can update to: https://github.com/sitespeedio/browsertime/tarball/89771a1d6be54114db190427dbc281582cba3d47
To test the upgrade, run a raptor test locally (with and without visual-metrics <code>--browsertime-visualmetrics</code> if possible) and test it on try with at least one test on desktop and mobile.
=== Finding the Geckodriver Being Used ===
If you're looking for the latest geckodriver being used there are two ways:
* Find the latest one from here: https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=geckodriver
* Alternatively, if you're trying to figure out which geckodriver a given CI task is using, you can click on the browsertime task in treeherder, and then click on the `Task` id in the bottom left of the pop-up interface. Then in the window that opens up, click on `See more` in the task details tab on the left, this will show you the dependent tasks with the latest toolchain-geckodriver being used. There's an Artifacts drop down on the right hand side for the toolchain-geckodriver task that you can find the latest geckodriver in.
If you're trying to test Browsertime with a new geckodriver, you can do either of the following:
* Request a new geckodriver build in your try run (i.e. through <code>./mach try fuzzy</code>).
* Trigger a new geckodriver in a try push, then trigger the browsertime tests which will then use the newly built version in the try push.
=== Comparing Before/After Browsertime Videos ===
We have some scripts that can produce side-by-side comparison videos for you of the worst pairing of videos. You can find the script here: https://github.com/gmierz/moz-current-tests#browsertime-side-by-side-video-comparisons
Once the side-by-side comparison is produced, the video on the left is the old/base video, and the video on the right is the new video.
=== Bugs ===
<bugzilla>
    {
"blocks": "1563082, 1561939"
    }
</bugzilla>
== WebExtension ==
=== Page-Load Tests ===
=== Page-Load Tests ===


Page-load tests involve loading a specific web page and measuring the load performance (i.e. [https://wiki.mozilla.org/TestEngineering/Performance/Glossary#First_Non-Blank_Paint_.28fnbpaint.29 time-to-first-non-blank-paint], [https://wiki.mozilla.org/TestEngineering/Performance/Glossary#First_Contentful_Paint_.28fcp.29 first-contentful-paint] , [https://wiki.mozilla.org/TestEngineering/Performance/Glossary#DOM_Content_Flushed_.28dcf.29 dom-content-flushed], [https://wiki.mozilla.org/TestEngineering/Performance/Glossary#Time_To_First_Interactive_.28ttfi.29 ttfi]).
Page-load tests involve loading a specific web page and measuring the load performance (i.e. [https://wiki.mozilla.org/TestEngineering/Performance/Glossary#First_Non-Blank_Paint_.28fnbpaint.29 time-to-first-non-blank-paint], first-contentful-paint, [https://wiki.mozilla.org/TestEngineering/Performance/Glossary#DOM_Content_Flushed_.28dcf.29 dom-content-flushed]).


For page-load tests by default, instead of using live web pages for performance testing, Raptor uses a tool called [[https://wiki.mozilla.org/TestEngineering/Performance/Raptor/Mitmproxy Mitmproxy]]. Mitmproxy allows us to record and playback test pages via a local Firefox proxy. The Mitmproxy recordings are stored on [https://github.com/mozilla/build-tooltool tooltool] and are automatically downloaded by Raptor when they are required for a test. Raptor uses mitmproxy via the [https://searchfox.org/mozilla-central/source/testing/mozbase/mozproxy mozbase mozproxy] package.
For page-load tests by default, instead of using live web pages for performance testing, Raptor uses a tool called [[/Mitmproxy/]]. Mitmproxy allows us to record and playback test pages via a local Firefox proxy. The Mitmproxy recordings are stored on [https://github.com/mozilla/build-tooltool tooltool] and are automatically downloaded by Raptor when they are required for a test. Raptor uses mitmproxy via the [https://searchfox.org/mozilla-central/source/testing/mozbase/mozproxy mozproxy] package.


There are two different types of Raptor page-load tests; warm page-load and cold page-load.
There are two different types of Raptor page-load tests: warm page-load and cold page-load.


==== Warm Page-Load ====
==== Warm Page-Load ====
For warm page-load tests, the desktop browser (or android browser app) is just started up once; so the browser is warm on each page-load.
For warm page-load tests, the browser is just started up once; so the browser is warm on each page-load.


'''Raptor warm page-load test process when running on Firefox/Chrome/Chromium desktop:'''
'''Raptor warm page-load test process when running on Firefox/Chrome/Chromium desktop:'''
Line 32: Line 175:
* The test URL is loaded; measurements taken
* The test URL is loaded; measurements taken
* The tab is reloaded 24 more times; measurements taken each time
* The tab is reloaded 24 more times; measurements taken each time
* The measurements from the first page-load are not included in overall results metrics b/c of first load noise; however they are listed in the perfherder-data.json/raptor.json artifacts
* The measurements from the first page-load are not included in overall results metrics b/c of first load noise; however they are listed in the JSON artifacts


'''Raptor warm page-load test process when running on Firefox android browser apps:'''
'''Raptor warm page-load test process when running on Firefox android browser apps:'''


* The android app data is cleared (via `adb shell pm clear firefox.app.binary.name`)
* The android app data is cleared (via <code>adb shell pm clear firefox.app.binary.name</code>)
* The new browser profile is copied onto the android device sdcard
* The new browser profile is copied onto the android device sdcard
* The Firefox android app is started up
* The Firefox android app is started up
* Post-startup browser settle pause of 30 seconds
* Post-startup browser settle pause of 30 seconds
* On Fennec only, a new browser tab is created (other Firefox apps use the single/existing tab)
* The test URL is loaded; measurements taken
* The test URL is loaded; measurements taken
* The tab is reloaded 14 more times; measurements taken each time
* The tab is reloaded 14 more times; measurements taken each time
* The measurements from the first page-load are not included in overall results metrics b/c of first load noise; however they are listed in the perfherder-data.json/raptor.json artifacts
* The measurements from the first page-load are not included in overall results metrics b/c of first load noise; however they are listed in the JSON artifacts


==== Cold Page-Load ====
==== Cold Page-Load ====
For cold page-load tests, the desktop browser (or android browser app) is shutdown and re-started between page load cycles; so the browser is cold on each page-load. This is what happens for Raptor cold page-load tests:
For cold page-load tests, the browser is shut down and restarted between page load cycles; so the browser is cold on each page-load. This is what happens for Raptor cold page-load tests:


'''Raptor cold page-load test process when running on Firefox/Chrome/Chromium desktop:'''
'''Raptor cold page-load test process when running on Firefox/Chrome/Chromium desktop:'''
Line 56: Line 198:
* The test URL is loaded; measurements taken
* The test URL is loaded; measurements taken
* The tab is closed
* The tab is closed
* The desktop browser is shutdown
* The desktop browser is shut down
* Entire process is repeated for the remaining browser cycles (25 cycles total)
* Entire process is repeated for the remaining browser cycles (25 cycles total)
* The measurements from all browser cycles are used to calculate overall results
* The measurements from all browser cycles are used to calculate overall results


'''Raptor cold page-load test process when running on Firefox android browser apps:'''
'''Raptor cold page-load test process when running on Firefox Android browser apps:'''


* The android app data is cleared (via `adb shell pm clear firefox.app.binary.name`)
* The Android app data is cleared (via <code>adb shell pm clear firefox.app.binary.name</code>)
* A new browser profile is created
* A new browser profile is created
* The new browser profile is copied onto the android device sdcard
* The new browser profile is copied onto the Android device sdcard
* The Firefox android app is started up
* The Firefox Android app is started up
* Post-startup browser settle pause of 30 seconds
* Post-startup browser settle pause of 30 seconds
* On Fennec only, a new browser tab is created (other Firefox apps use the single/existing tab)
* The test URL is loaded; measurements taken
* The test URL is loaded; measurements taken
* The android app is shutdown
* The Android app is shut down
* Entire process is repeated for the remaining browser cycles (15 cycles total)
* Entire process is repeated for the remaining browser cycles (15 cycles total)
* Note that the SSL cert DB is only created once (browser cycle 1) and copied into the profile for each additional browser cycle; thus not having to use the 'certutil' tool and re-created the db on each cycle
* Note that the SSL cert DB is only created once (browser cycle 1) and copied into the profile for each additional browser cycle; thus not having to use the 'certutil' tool and re-created the db on each cycle
Line 75: Line 216:


==== Using Live Sites ====
==== Using Live Sites ====
It is possible to use live web pages for the page-load tests instead of using the mitproxy recordings. This option is available when running on Try only; as we don't want to submit data from live pages to Perfherder (since live page content will always be changing).
It is possible to use live web pages for the page-load tests instead of using the mitmproxy recordings. To do this, add <code>--live</code> to your command line or select one of the 'live' variants when running <code>./mach try fuzzy --full</code>.


To run a particular Raptor tp6 page-load test with live sites, open the raptor-tp6*.ini file ([https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests Raptor tests folder]), and for the test default (or under a single page/subtest) just add this attribute:
==== Disabling Alerts ====
It is possible to disable alerting for all our performance tests. Open the target test manifest such as the raptor-tp6*.ini file ([https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests Raptor tests folder]), and make sure there are no <tt>alert_on</tt> specifications.


  use_live_sites = true
When it's removed there will no longer be a <tt>shouldAlert</tt> field in the output Perfherder data (you can find the [https://searchfox.org/mozilla-central/source/testing/mozharness/external_tools/performance-artifact-schema.json#68,165 schema here]). As long as <tt>shouldAlert</tt> is not in the data, no alerts will be generated. If you need to also disable code sheriffing for the test, then you need to change the tier of the task to 3.


And push that change to Try (./mach try fuzzy --full) and run the Raptor page-load test.
==== High value tests ====
 
We have a notion of '''high-value''' tests in performance testing. These are chosen based on how many alerts they can catch using [https://github.com/gmierz/moz-current-tests/blob/master/high-value-tests/generate_high_value_tests.py this script] along with [https://github.com/gmierz/moz-current-tests/blob/master/high-value-tests/sql_query.txt a redash query]. The lists below are the minimum set of test pages we should run to catch as many alerts as we can.
 
'''Desktop'''
 
Last updated: June 2021
 
    amazon
    bing
    cnn
    fandom
    gslides
    instagram
    twitter
    wikipedia
    yahoo-mail
 
'''Mobile'''
 
Last updated: November 2020
 
    allrecipes
    amazon-search
    espn
    facebook
    google-search
    microsoft-support
    youtube-watch


=== Benchmark Tests ===
=== Benchmark Tests ===
Line 93: Line 263:
# '''memory-usage tests'''
# '''memory-usage tests'''
# '''CPU-usage tests'''
# '''CPU-usage tests'''
For a combined-measurement run with distinct Perfherder output for each measurement type, you can do:
  ./mach raptor --test raptor-scn-power-idle-bg-fenix --app fenix --binary org.mozilla.fenix.performancetest --host 10.0.0.16 --power-test --memory-test --cpu-test
Each measurement subtype (power-, memory-, and cpu-usage) will have a corresponding PERFHERDER_DATA blob:
<pre>22:31:05    INFO -  raptor-output Info: PERFHERDER_DATA: {"framework": {"name": "raptor"}, "suites": [{"name": "raptor-scn-power-idle-bg-fenix-cpu", "lowerIsBetter": true, "alertThreshold": 2.0, "value": 0, "subtests": [{"lowerIsBetter": true, "unit": "%", "name": "cpu-browser_cpu_usage", "value": 0, "alertThreshold": 2.0}], "type": "cpu", "unit": "%"}]}
22:31:05    INFO -  raptor-output Info: cpu results can also be found locally at: /Users/sdonner/moz_src/mozilla-unified/testing/mozharness/build/raptor-cpu.json
</pre>
(repeat for power, memory snippets)


==== Power-Use Tests (Android) ====
==== Power-Use Tests (Android) ====
Line 108: Line 289:
* We set `scenario_time` to '''20 minutes''' (1200000 milliseconds), and `page_timeout` to '22 minutes' (1320000 milliseconds)
* We set `scenario_time` to '''20 minutes''' (1200000 milliseconds), and `page_timeout` to '22 minutes' (1320000 milliseconds)
** It's crucial that `page_timeout` exceed `scenario_time`; if not, measurement tests will fail/bail early
** It's crucial that `page_timeout` exceed `scenario_time`; if not, measurement tests will fail/bail early
* We launch the {Fenix, Fennec, GeckoView, Reference Browser} on-Android app
* We launch the {Fenix, GeckoView, Reference Browser} on-Android app
* Post-startup browser settle pause of 30 seconds
* Post-startup browser settle pause of 30 seconds
* On Fennec only, a new browser tab is created (other Firefox apps use the single/existing tab)
* On Fennec only, a new browser tab is created (other Firefox apps use the single/existing tab)
Line 115: Line 296:
* Raw power-use measurement data is listed in the perfherder-data.json/raptor.json artifacts
* Raw power-use measurement data is listed in the perfherder-data.json/raptor.json artifacts


In the Perfherder (or Firefox Health) dashboards for these power usage tests, all data points have milli-Ampere-hour units, with a lower value being better.
In the Perfherder dashboards for these power usage tests, all data points have milli-Ampere-hour units, with a lower value being better.
Proportional power usage is the total power usage of hidden battery sippers that is proportionally "smeared"/distributed across all open applications.
Proportional power usage is the total power usage of hidden battery sippers that is proportionally "smeared"/distributed across all open applications.


Line 121: Line 302:


To run on a tethered phone via USB from a macOS host, on:
To run on a tethered phone via USB from a macOS host, on:
===== Fennec =====
  ./mach raptor --test raptor-scn-power-idle-fennec --app fennec --binary org.mozilla.firefox --power-test --host 10.252.27.96


===== Fenix =====
===== Fenix =====
Line 148: Line 325:
* pgo builds tend to be -- from my limited empirical evidence -- about 10 - 15 minutes longer to complete than their opt counterparts
* pgo builds tend to be -- from my limited empirical evidence -- about 10 - 15 minutes longer to complete than their opt counterparts


==== Perf Dashboards ====
==== Dashboards ====


* Perfherder example (GeckoView): https://treeherder.mozilla.org/perf.html#/graphs?timerange=2592000&series=mozilla-central,2027286,1,10&series=mozilla-central,2027291,1,10&series=mozilla-central,2027296,1,10
See [https://wiki.mozilla.org/TestEngineering/Performance/Results performance results] for our various dashboards.
* [https://github.com/mozilla-frontend-infra/firefox-health-dashboard/issues/420 Coming soon] to https://health.graphics/android
 
==== Comparing Run Results ====
Coming soon: comparing local runs to CI results


=== Running Locally ===
=== Running Locally ===
Line 210: Line 383:


* Device is in 'superuser' mode
* Device is in 'superuser' mode
** [stephend] - I want to explain this a bit more, so leaving this comment as a reminder


* The geckoview example app is already installed on the device (from ./mach bootstrap, above). Download the geckoview_example.apk from the appropriate [https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=android%2Cbuild android build on treeherder], then install it on your device, i.e.:
* The geckoview example app is already installed on the device (from ./mach bootstrap, above). Download the geckoview_example.apk from the appropriate [https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=android%2Cbuild android build on treeherder], then install it on your device, i.e.:
Line 284: Line 456:


With that setting, Raptor will not start the playback tool (i.e. Mitmproxy) and will not turn on the corresponding browser proxy, therefore forcing the test page to load live.
With that setting, Raptor will not start the playback tool (i.e. Mitmproxy) and will not turn on the corresponding browser proxy, therefore forcing the test page to load live.
When `use_live_pages = true` and a page-load test is measuring hero element (set in the test INI 'measure' option) then the hero element measurement will automatically be dropped - because the hero elements only exist in our Mitmproxy recordings and not in live pages.
The word 'live' will be appended to the test name in the PERFHERDER_DATA so live sites can be specifically seen in perfherder for try runs.
'''Important:''' This is fine for running on try, but we don't want to enable live sites in the production repos - because we don't want live site data being ingested by perfherder and used for regression alerting etc. Therefore as a safety catch, if using live sites the test won't even run unless running locally or on try.


=== Running Raptor on Try ===
=== Running Raptor on Try ===
Line 311: Line 477:
==== Raptor Hardware in Production ====
==== Raptor Hardware in Production ====


The Raptor performance tests run on dedicated hardware (the same hardware that the Talos performance tests use). See the [[https://wiki.mozilla.org/Performance_sheriffing/Talos/Misc#Hardware_Profile_of_machines_used_in_automation|Talos hardware used in automation wiki page]] for more details.
The Raptor performance tests run on dedicated hardware (the same hardware that the Talos performance tests use). See the [[TestEngineering/Performance/Platforms|performance platforms]] for more details.


=== Profiling Raptor Jobs ===
=== Profiling Raptor Jobs ===


Raptor tests are able to create Gecko profiles which can be viewed in [https://perf-html.io/ perf-html.io.] This is currently only supported when running Raptor on Firefox desktop.
Raptor tests are able to create Gecko profiles which can be viewed in [https://profiler.firefox.com/ profiler.firefox.com.] This is currently only supported when running Raptor on Firefox desktop.


==== Nightly Profiling Jobs in Production ====
==== Nightly Profiling Jobs in Production ====
Line 332: Line 498:
Note: While profiling is turned on, Raptor will automatically reduce the number of pagecycles to 3. If you wish to override this, add the --page-cycles argument to the raptor command line.  
Note: While profiling is turned on, Raptor will automatically reduce the number of pagecycles to 3. If you wish to override this, add the --page-cycles argument to the raptor command line.  


Raptor will automatically launch Firefox and load the latest Gecko profile in [https://perf-html.io perfhtml.io]. To turn this feature off, just set the DISABLE_PROFILE_LAUNCH=1 env var.
Raptor will automatically launch Firefox and load the latest Gecko profile in [https://profiler.firefox.com profiler.firefox.com]. To turn this feature off, just set the DISABLE_PROFILE_LAUNCH=1 env var.


If auto-launch doesn't work for some reason, just start Firefox manually and browse to [https://perf-html.io perfhtml.io], click on "Browse" and select the Raptor profile ZIP file noted above.
If auto-launch doesn't work for some reason, just start Firefox manually and browse to [https://profiler.firefox.com profiler.firefox.com], click on "Browse" and select the Raptor profile ZIP file noted above.


If you're on Windows and want to profile a Firefox build that you compiled yourself, make sure it contains profiling information and you have a symbols zip for it, by following the [https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Profiling_with_the_Built-in_Profiler_and_Local_Symbols_on_Windows#Profiling_local_talos_runs directions on MDN].
If you're on Windows and want to profile a Firefox build that you compiled yourself, make sure it contains profiling information and you have a symbols zip for it, by following the [https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Profiling_with_the_Built-in_Profiler_and_Local_Symbols_on_Windows#Profiling_local_talos_runs directions on MDN].
Line 375: Line 541:
# Click on the 'Job Details' tab below
# Click on the 'Job Details' tab below
# The Raptor profile ZIP files will be listed as job artifacts;
# The Raptor profile ZIP files will be listed as job artifacts;
# Select a Raptor profile ZIP artifact, and click the 'view in perf-html.io' link to the right
# Select a Raptor profile ZIP artifact, and click the 'view in Firefox Profiler' link to the right


=== Recording Pages for Raptor Pageload Tests ===
=== Recording Pages for Raptor Pageload Tests ===
Line 408: Line 574:
=== Page-Load Tests ===
=== Page-Load Tests ===


For all Raptor page-load tests, the pages are played back from [[https://wiki.mozilla.org/TestEngineering/Performance/Raptor/Mitmproxy Mitmproxy]] recordings. If you need the HTML page source (outside of the Mitmproxy recording) for debugging, the raw HTML can be found in our [https://github.com/mozilla/perf-automation/tree/master/pagesets perf-automation github repo].
Raptor page-load test documentation is generated by [https://firefox-source-docs.mozilla.org/code-quality/lint/linters/perfdocs.html PerfDocs] and available in the [https://firefox-source-docs.mozilla.org/testing/perfdocs/raptor.html Firefox Source Docs].
 
All the pages in a test suite an be run by calling the top-level test name, i.e.:
 
  ./mach raptor --test raptor-tp6-1
 
Individual test pages can be ran by calling the subtest, i.e.:
 
  ./mach raptor --test raptor-tp6-google-firefox
 
Some of the page recordings contain [[https://wiki.mozilla.org/Performance_sheriffing/Raptor/Mitmproxy#Adding_Hero_Elements hero elements]]. When hero elements are measured, the value is the time until the hero element appears on the page (in MS).
 
Below are the details for each page-load suite, and the test pages contained within each.
 
==== raptor-tp6-1 ====
* contact: :rwood, :jmaher
* type: page-load
* browsers: Firefox desktop, Google Chrome desktop (Windows and OSX only at the moment)
* measuring on Firefox: time-to-first-non-blank-paint, first-contentful-paint, dom-content-flushed, hero element, time-to-first-interactive, loadtime
* measuring on Chrome: first-contentful-paint, hero element, loadtime
* page-cycles: 25
* reporting: Each pagecycle measures all the values (in MS). The first page-cycle is dropped due to the initial extra loading time/noise. The overall result reported for each test page is the median of the values reported for each pagecycle (in MS).
* test INI: [https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/raptor-tp6-1.ini raptor-tp6-1.ini].
 
''' Test pages in tp6-1 (* = firefox or chrome):'''
 
[raptor-tp6-amazon-*]
* URL: https://www.amazon.com/s/url=search-alias%3Daps&field-keywords=laptop
* Hero: string description element for first laptop in search results
 
[raptor-tp6-facebook-*]
* URL: https://www.facebook.com (logged into a user account)
* Hero: on the Facebook 'Home' icon
 
[raptor-tp6-google-*]
* URL: https://www.google.com/search?hl=en&q=barack+obama&cad=h
* Hero: bigger photo of Obama in search results towards top right
 
[raptor-tp6-youtube-*]
* URL: https://www.youtube.com
* Hero: YouTube logo on the top left
 
==== raptor-tp6-2 ====
* contact: :rwood, :jmaher
* type: page-load
* browsers: Firefox desktop, Google Chrome desktop (Windows and OSX only at the moment)
* measuring on Firefox: time-to-first-non-blank-paint, first-contentful-paint, dom-content-flushed, hero element, time-to-first-interactive, loadtime
* measuring on Chrome: first-contentful-paint, hero element, loadtime
* page-cycles: 25
* reporting: Each pagecycle measures all the values (in MS). The first page-cycle is dropped due to the initial extra loading time/noise. The overall result reported for each test page is the median of the values reported for each pagecycle (in MS).
* test INI: [https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/raptor-tp6-2.ini raptor-tp6-2.ini].
 
''' Test pages in tp6-2 (* = firefox or chrome):'''
 
[raptor-tp6-docs-*]
* URL: https://docs.google.com/document/d/1US-07msg12slQtI_xchzYxcKlTs6Fp7WqIc6W5GK5M8/edit?usp=sharing
* Hero: blue 'Sign In' button on the top right
 
[raptor-tp6-sheets-*]
* URL: https://docs.google.com/spreadsheets/d/1jT9qfZFAeqNoOK97gruc34Zb7y_Q-O_drZ8kSXT-4D4/edit?usp=sharing
* Hero: blue 'Sign In' button on the top right
 
[raptor-tp6-slides-*]
* URL: https://docs.google.com/presentation/d/1Ici0ceWwpFvmIb3EmKeWSq_vAQdmmdFcWqaiLqUkJng/edit?usp=sharing
* Hero: blue 'Sign In' button on the top right
 
==== raptor-tp6-3 ====
* contact: :rwood, :jmaher, :bebe
* type: page-load
* browsers: Firefox desktop, Google Chrome desktop (Windows and OSX only at the moment)
* measuring on Firefox: time-to-first-non-blank-paint, first-contentful-paint, dom-content-flushed, time-to-first-interactive, loadtime
* measuring on Chrome: first-contentful-paint, loadtime
* page-cycles: 25
* reporting: Each pagecycle measures all the values (in MS). The first page-cycle is dropped due to the initial extra loading time/noise. The overall result reported for each test page is the median of the values reported for each pagecycle (in MS).
* test INI: [https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/raptor-tp6-3.ini raptor-tp6-3.ini].
 
''' Test pages in tp6-3 (* = firefox or chrome):'''
 
[raptor-tp6-imdb-*]
* URL: https://www.imdb.com/title/tt0084967/?ref_=nv_sr_2
 
[raptor-tp6-imgur-*]
* URL: https://imgur.com/gallery/m5tYJL6
 
[raptor-tp6-wikia-*]
* URL: http://fandom.wikia.com/articles/fallout-76-will-live-and-die-on-the-creativity-of-its-playerbase
 
==== raptor-tp6-4 ====
* contact: :rwood, :jmaher, :bebe
* type: page-load
* browsers: Firefox desktop, Google Chrome desktop (Windows and OSX only at the moment)
* measuring on Firefox: time-to-first-non-blank-paint, first-contentful-paint, dom-content-flushed, time-to-first-interactive, loadtime
* measuring on Chrome: first-contentful-paint, loadtime
* page-cycles: 25
* reporting: Each pagecycle measures all the values (in MS). The first page-cycle is dropped due to the initial extra loading time/noise. The overall result reported for each test page is the median of the values reported for each pagecycle (in MS).
* test INI: [https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/raptor-tp6-4.ini raptor-tp6-4.ini].
 
''' Test pages in tp6-4 (* = firefox or chrome):'''
 
[raptor-tp6-bing-*]
* URL: https://www.bing.com/search?q=barack+obama
 
[raptor-tp6-yandex-*]
* URL: https://yandex.ru/search/?text=barack%20obama&lr=10115
 
==== raptor-tp6-5 ====
* contact: :rwood, :jmaher, :bebe
* type: page-load
* browsers: Firefox desktop, Google Chrome desktop (Windows and OSX only at the moment)
* measuring on Firefox: time-to-first-non-blank-paint, first-contentful-paint, dom-content-flushed, time-to-first-interactive, loadtime
* measuring on Chrome: first-contentful-paint, loadtime
* page-cycles: 25
* reporting: Each pagecycle measures all the values (in MS). The first page-cycle is dropped due to the initial extra loading time/noise. The overall result reported for each test page is the median of the values reported for each pagecycle (in MS).
* test INI: [https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/raptor-tp6-5.ini raptor-tp6-5.ini].
 
''' Test pages in tp6-5 (* = firefox or chrome):'''
 
[raptor-tp6-apple-*]
* URL: https://www.apple.com/macbook-pro/
 
[raptor-tp6-microsoft-*]
* URL: https://www.microsoft.com/en-us/windows/get-windows-10
 
==== raptor-tp6-6 ====
* contact: :rwood, :jmaher, :bebe
* type: page-load
* browsers: Firefox desktop, Google Chrome desktop (Windows and OSX only at the moment)
* measuring on Firefox: time-to-first-non-blank-paint, first-contentful-paint, dom-content-flushed, time-to-first-interactive, loadtime
* measuring on Chrome: first-contentful-paint, loadtime
* page-cycles: 25
* reporting: Each pagecycle measures all the values (in MS). The first page-cycle is dropped due to the initial extra loading time/noise. The overall result reported for each test page is the median of the values reported for each pagecycle (in MS).
* test INI: [https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/raptor-tp6-6.ini raptor-tp6-6.ini].
 
''' Test pages in tp6-6 (* = firefox or chrome):'''
 
[raptor-tp6-reddit-*]
* URL: https://www.reddit.com/r/technology/comments/9sqwyh/we_posed_as_100_senators_to_run_ads_on_facebook/
 
[raptor-tp6-yahoo-news-*]
* URL: https://www.yahoo.com/lifestyle/police-respond-noise-complaint-end-playing-video-games-respectful-tenants-002329963.html
 
==== raptor-tp6-7 ====
* contact: :rwood, :jmaher, :bebe
* type: page-load
* browsers: Firefox desktop, Google Chrome desktop (Windows and OSX only at the moment)
* measuring on Firefox: time-to-first-non-blank-paint, first-contentful-paint, dom-content-flushed, time-to-first-interactive, loadtime
* measuring on Chrome: first-contentful-paint, loadtime
* page-cycles: 25
* reporting: Each pagecycle measures all the values (in MS). The first page-cycle is dropped due to the initial extra loading time/noise. The overall result reported for each test page is the median of the values reported for each pagecycle (in MS).
* test INI: [https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/raptor-tp6-7.ini raptor-tp6-7.ini].
 
''' Test pages in tp6-7 (* = firefox or chrome):'''
 
[raptor-tp6-instagram-*]
* URL: https://www.instagram.com/
 
[raptor-tp6-twitter-*]
* URL: https://twitter.com/BarackObama
 
[raptor-tp6-yahoo-mail-*]
* URL: https://mail.yahoo.com/
 
==== raptor-tp6-8 ====
* contact: :rwood, :jmaher, :bebe
* type: page-load
* browsers: Firefox desktop, Google Chrome desktop (Windows and OSX only at the moment)
* measuring on Firefox: time-to-first-non-blank-paint, first-contentful-paint, dom-content-flushed, time-to-first-interactive, loadtime
* measuring on Chrome: first-contentful-paint, loadtime
* page-cycles: 25
* reporting: Each pagecycle measures all the values (in MS). The first page-cycle is dropped due to the initial extra loading time/noise. The overall result reported for each test page is the median of the values reported for each pagecycle (in MS).
* test INI: [https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/raptor-tp6-8.ini raptor-tp6-8.ini].
 
''' Test pages in tp6-8 (* = firefox or chrome):'''
 
[raptor-tp6-ebay-*]
* URL: https://www.ebay.com/
 
[raptor-tp6-wikipedia-*]
* URL: https://en.wikipedia.org/wiki/Barack_Obama
 
==== raptor-tp6-9 ====
* contact: :rwood, :jmaher, :bebe
* type: page-load
* browsers: Firefox desktop, Google Chrome desktop (Windows and OSX only at the moment)
* measuring on Firefox: time-to-first-non-blank-paint, first-contentful-paint, dom-content-flushed, time-to-first-interactive, loadtime
* measuring on Chrome: first-contentful-paint, loadtime
* page-cycles: 25
* reporting: Each pagecycle measures all the values (in MS). The first page-cycle is dropped due to the initial extra loading time/noise. The overall result reported for each test page is the median of the values reported for each pagecycle (in MS).
* test INI: [https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/raptor-tp6-9.ini raptor-tp6-9.ini].
 
''' Test pages in tp6-9 (* = firefox or chrome):'''
 
[raptor-tp6-google-mail-*]
* URL: https://mail.google.com/
 
[raptor-tp6-pinterest-*]
* URL: https://pinterest.com/
 
==== raptor-tp6-10 ====
* contact: :rwood, :jmaher, :bebe
* type: page-load
* browsers: Firefox desktop, Google Chrome desktop (Windows and OSX only at the moment)
* measuring on Firefox: time-to-first-non-blank-paint, first-contentful-paint, dom-content-flushed, time-to-first-interactive, loadtime
* measuring on Chrome: first-contentful-paint, loadtime
* page-cycles: 25
* reporting: Each pagecycle measures all the values (in MS). The first page-cycle is dropped due to the initial extra loading time/noise. The overall result reported for each test page is the median of the values reported for each pagecycle (in MS).
* test INI: [https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/raptor-tp6-10.ini raptor-tp6-10.ini].
 
''' Test pages in tp6-10 (* = firefox or chrome):'''
 
[raptor-tp6-paypal-*]
* URL: https://www.paypal.com/myaccount/summary/
 
==== raptor-tp6m-1 ====
* contact: :rwood, :davehunt
* type: page-load
* browsers: Firefox Android Geckoview Example App
* measuring: time-to-first-non-blank-paint, first-contentful-paint, dom-content-flushed, time-to-first-interactive, loadtime
* page-cycles: 15
* reporting: Each pagecycle measures all the values (in MS). The first page-cycle is dropped due to the initial extra loading time/noise. The overall result reported for each test page is the median of the values reported for each pagecycle (in MS).
* test INI: [https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/raptor-tp6m-1.ini raptor-tp6m-1.ini].
 
''' Test pages in tp6m-1 (* = firefox or chrome):'''
 
[raptor-tp6m-amazon-*]
* URL: https://www.amazon.com
* Hero: None
 
[raptor-tp6m-facebook-*]
* URL: https://m.facebook.com (logged into a user account)
* Hero: None
 
[raptor-tp6m-google-*]
* URL: https://www.google.com
* Hero: None
 
[raptor-tp6m-youtube-*]
* URL: https://www.youtube.com
* Hero: None


=== Benchmark Tests ===
=== Benchmark Tests ===
 
{{TODO|Move these to PerfDocs}}
==== raptor-assorted-dom ====
==== assorted-dom ====
* contact: ?
* contact: ?
* type: benchmark
* type: benchmark
Line 656: Line 584:
* TODO
* TODO


==== raptor-motionmark-animometer, raptor-motionmark-htmlsuite ====
==== motionmark-animometer, motionmark-htmlsuite ====
* contact: ?
* contact: ?
* type: benchmark
* type: benchmark
Line 665: Line 593:
** suite: we take a geometric mean of all the subtests (9 for animometer, 11 for html suite)
** suite: we take a geometric mean of all the subtests (9 for animometer, 11 for html suite)


==== raptor-speedometer ====
==== speedometer ====
* contact: :selena
* contact: :selena
* type: benchmark
* type: benchmark
Line 673: Line 601:
* data: there are 16 subtests in Speedometer; each of these are made up of 9 internal benchmarks.
* data: there are 16 subtests in Speedometer; each of these are made up of 9 internal benchmarks.
* summarization:
* summarization:
** subtest: For all of the 16 subtests, we collect the sum of all their internal benchmark results.
** subtest: For all of the 16 subtests, we collect [https://searchfox.org/mozilla-central/source/third_party/webkit/PerformanceTests/Speedometer/resources/benchmark-report.js#66-67 a summed of all their internal benchmark results] for each of them. To obtain a single score per subtest, we take [https://searchfox.org/mozilla-central/source/testing/raptor/raptor/output.py#427-470 a median of the replicates].
** score: geometric mean of the 16 sums
** score: [https://searchfox.org/mozilla-central/source/testing/raptor/raptor/output.py#317-330 geometric mean of the 16 subtest metrics (along with some special corrections)].


This is the [http://browserbench.org/Speedometer/ Speedometer] JavaScript benchmark taken verbatim and slightly modified to work with the Raptor harness.
This is the [http://browserbench.org/Speedometer/ Speedometer v1.0] JavaScript benchmark taken verbatim and slightly modified to work with the Raptor harness.


==== raptor-stylebench ====
==== stylebench ====
* contact: :emilio
* contact: :emilio
* type: benchmark
* type: benchmark
Line 685: Line 613:
* reporting: runs/minute score
* reporting: runs/minute score


==== raptor-sunspider ====
==== sunspider ====
* contact: ?
* contact: ?
* type: benchmark
* type: benchmark
Line 691: Line 619:
* TODO
* TODO


==== raptor-unity-webgl ====
==== unity-webgl ====
* contact: ?
* contact: ?
* type: benchmark
* type: benchmark
Line 697: Line 625:
* TODO
* TODO


==== raptor-youtube-playback ====
==== youtube-playback ====
* contact: ?
* contact: ?
* type: benchmark
* type: benchmark
* details: https://wiki.mozilla.org/Performance_sheriffing/Raptor/Youtube_playback_performance
* details: [[/Youtube_playback_performance|YouTube playback performance]]
* browsers: Firefox desktop, Firefox Android Geckoview
* browsers: Firefox desktop, Firefox Android Geckoview
* measuring: media streaming playback performance (dropped video frames)
* measuring: media streaming playback performance (dropped video frames)
Line 709: Line 637:
This are the [https://ytlr-cert.appspot.com/2019/main.html?test_type=playbackperf-test Playback Performance Tests] benchmark taken verbatim and slightly modified to work with the Raptor harness.
This are the [https://ytlr-cert.appspot.com/2019/main.html?test_type=playbackperf-test Playback Performance Tests] benchmark taken verbatim and slightly modified to work with the Raptor harness.


==== raptor-wasm-misc, raptor-wasm-misc-baseline, raptor-wasm-misc-ion ====
==== wasm-misc, wasm-misc-baseline, wasm-misc-ion ====
* contact: ?
* contact: ?
* type: benchmark
* type: benchmark
Line 715: Line 643:
* TODO
* TODO


==== raptor-wasm-godot, raptor-wasm-godot-baseline, raptor-wasm-godot-ion  ====
==== wasm-godot, wasm-godot-baseline, wasm-godot-ion  ====
* contact: ?
* contact: ?
* type: benchmark
* type: benchmark
Line 721: Line 649:
* TODO
* TODO


==== raptor-webaudio ====
==== webaudio ====
* contact: ?
* contact: ?
* type: benchmark
* type: benchmark
Line 749: Line 677:
* browsers: Android: Fennec 64.0.2, GeckoView Example, Fenix, and Reference Browser
* browsers: Android: Fennec 64.0.2, GeckoView Example, Fenix, and Reference Browser
* measuring: Power consumption for idle Android browsers, with about:blank loaded and app backgrounded, over a 10-minute duration
* measuring: Power consumption for idle Android browsers, with about:blank loaded and app backgrounded, over a 10-minute duration
== Debugging Desktop Product Failures ==
As of now, there is no easy way to do this. Raptor was not built for debugging functional failures. Hitting these in Raptor is indicative that we lack functional test coverage so regression tests should be added for those failures after they are fixed.
To debug a functional failure in Raptor you can follow these steps:
# If bug 1653617 has not landed yet, apply the patch.
# Add the --verbose flag to the extra-options list [https://searchfox.org/mozilla-central/source/taskcluster/ci/test/raptor.yml#98-101 here].
# If the --setenv doesn't exist yet ([https://bugzilla.mozilla.org/show_bug.cgi?id=1494669 bug 1494669]), then add your MOZ_LOG environment variables to give you additional logging [https://searchfox.org/mozilla-central/source/testing/raptor/raptor/webextension/desktop.py#42 here].
# If the flag does exist, then you can add the MOZ_LOG variables to the [https://searchfox.org/mozilla-central/source/taskcluster/ci/test/raptor.yml raptor.yml] configuration file.
# Push to try if you can't reproduce the failure locally.
You can follow [https://bugzilla.mozilla.org/show_bug.cgi?id=1655554 bug 1655554] as we work on improving this workflow.
In some cases, you might not be able to get logging for what you are debugging (browser console logging for instance). In this case, you should make your own debug prints with printf or something along those lines ([https://matrix.to/#/!LfXZSWEroPFPMQcYmw:mozilla.org/$r_azj7OipkgDzQ75SCns2QIayp4260PIMHLWLApJJNg?via=mozilla.org&via=matrix.org&via=rduce.org see :agi's debugging work for an example]).


== Debugging the Raptor Web Extension ==
== Debugging the Raptor Web Extension ==
Line 849: Line 792:
! Project !! Repository !! Tests results !! Schedule
! Project !! Repository !! Tests results !! Schedule
|-
|-
| Fenix (aka Firefox Preview) || [https://github.com/mozilla-mobile/fenix/ Github] || [https://treeherder.mozilla.org/#/jobs?repo=fenix Treeherder view] || Every 24 hours [https://tools.taskcluster.net/hooks/project-mobile/fenix-raptor Taskcluster Hook]
| Fenix || [https://github.com/mozilla-mobile/fenix/ Github] || [https://treeherder.mozilla.org/#/jobs?repo=fenix Treeherder view] || Every 24 hours [https://tools.taskcluster.net/hooks/project-releng/cron-task-mozilla-mobile-fenix%2Fraptor Taskcluster force hook]
|-
|-
| Reference-Browser || [https://github.com/mozilla-mobile/reference-browser/ Github] || [https://treeherder.mozilla.org/#/jobs?repo=reference-browser Treeherder view] || On demand [https://tools.taskcluster.net/hooks/project-mobile/reference-browser-raptor Taskcluster Hook]
| Reference-Browser || [https://github.com/mozilla-mobile/reference-browser/ Github] || [https://treeherder.mozilla.org/#/jobs?repo=reference-browser Treeherder view] || On each push
|}
|}


Tests are defined differently from what exists in mozilla-central. Taskcluster payloads are expressed in Python function in:
Tests are now defined in a similar fashion compared to what exists in mozilla-central. Task definitions are expressed in Yaml:
* https://github.com/mozilla-mobile/reference-browser/blob/f2ae31e23e36a749b937ff9728c28d53760242eb/automation/taskcluster/lib/tasks.py#L478-L616
* https://github.com/mozilla-mobile/fenix/blob/1c9c5317eb33d92dde3293dfe6a857c279a7ab12/taskcluster/ci/raptor/kind.yml
* https://github.com/mozilla-mobile/fenix/blob/8928822e99ff09ab45bce8ebab63aead10b7ebde/automation/taskcluster/lib/tasks.py#L455-L561
* https://github.com/mozilla-mobile/reference-browser/blob/4560a83cb559d3d4d06383205a8bb76a44336704/taskcluster/ci/raptor/kind.yml
 
Once defined, you must call these functions:
* https://github.com/mozilla-mobile/reference-browser/blob/f2ae31e23e36a749b937ff9728c28d53760242eb/automation/taskcluster/decision_task.py#L83-L96
* https://github.com/mozilla-mobile/fenix/blob/8928822e99ff09ab45bce8ebab63aead10b7ebde/automation/taskcluster/decision_task.py#L82-L91


If you want to test your changes on a PR, before they land, you need to apply a patch like this one: https://github.com/mozilla-mobile/fenix/commit/4cc16d4268240393f57b3711ab423c2407aeffb7. Don't forget to revert it before merging the patch.  
If you want to test your changes on a PR, before they land, you need to apply a patch like this one: https://github.com/mozilla-mobile/fenix/pull/5565/files. Don't forget to revert it before merging the patch.  


On Fenix and Reference-Browser, the raptor revision is tied to the latest nightly of mozilla-central  
On Fenix and Reference-Browser, the raptor revision is tied to the latest nightly of mozilla-central  


For more information, please reach out to :jlorenzo or :mhentges in #cia
For more information, please reach out to :jlorenzo or :mhentges in #cia
== Code formatting on Raptor ==
As Raptor is a Mozilla project we fallow the general Python coding style:
* https://firefox-source-docs.mozilla.org/tools/lint/coding-style/coding_style_python.html
[https://github.com/psf/black/ black] is the tool used to reformat the Python code.
Retrieved from "https://wiki.allizom.org/Special:MobileDiff/1216938...1236932"