TestEngineering/Performance/Talos/Tests: Difference between revisions

There are two different species of Talos tests:

In addition we have some variations on existing tests:

Some tests measure different things:

* [[#Benchmarks]]: These are benchmarks that measure specific items and report a summarized score

[https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/startup_test Startup tests] launch Firefox and measure the time to the onload or paint events. We run this in a series of cycles (default to 20) to generate a full set of data.  Tests that currently are startup tests are:

Many of the talos tests use the page loader to load a manifest of pages.

These are tests that load a specific page and measure the time it takes to load the page, scroll the page, draw the page etc.  In order to run a page load test, you need a manifest of pages to run.  The manifest is simply a list of URLs of pages to load, separated by carriage returns, e.g.:

</pre>

All our testing is done with empty blank profiles, this is not ideal for finding issues for end users.  We recently undertook a task to create a daily update to a profile so it is modern and relevant.  It browses a variety of web pages, and have history and cache to give us a more realistic scenario.

Currently we have issues with this on windows (takes too long to unpack the files from the profile), so we have turned this off there.  Our goal is to run this on basic pageload and startup tests.

Web Extensions are what Firefox has switched to and there are different code paths and APIs used vs addons.  Historically we don't test with addons (other than our test addons) and are missing out on common slowdowns.  In 2017 we started running some startup and basic pageload tests with a web extension in the profile ({{bug|1398974}}).  We have updated the Extension to be more real world and will continue to do that.

Paint tests are measuring the time to receive both the [https://developer.mozilla.org/en-US/docs/Web/Events/MozAfterPaint MozAfterPaint] and OnLoad event instead of just the OnLoad event.  Most tests now look for this unless they are an ASAP test, or an internal benchmark

We have a variety of tests which we now run in ASAP mode where we render as fast as possible (disabling vsync and letting the rendering iterate as fast as it can using `requestAnimationFrame`).  In fact we have replaced some original tests with the 'x' versions to make them measure.  We do this with RequestAnimationFrame().

Many tests have internal benchmarks which we report as accurately as possible.  These are the exceptions to the general rule of calculating the suite score as a geometric mean of the subtest values (which are median values of the raw data from the subtests).

Tests which are imported benchmarks are:

To get more stable numbers, tests are run multiple times. There are two ways that we do this: row major and column major. Row major means each test is run multiple times and then we move to the next test (and run it multiple times). Column major means that each test is run once one after the other and then the whole sequence of tests is run again.

More background information about these approaches can be found in Joel Maher's [https://elvis314.wordpress.com/2012/03/12/reducing-the-noise-in-talos/ Reducing the Noise in Talos] blog post.

We run our tests 100% offline, but serve pages via a webserver.  Knowing this we need to store and make available the offline pages we use for testing.

Some tests make use of a set of 50 "real world" pages, known as the tp5n set. These pages are not part of the talos repository, but without them the tests which use them won't run.

* To add these pages to your local setup, download [https://github.com/rwood-moz/talos-pagesets/tp5n.zip tp5n.zip], and extract it such that `'''tp5n'''` ends up as `testing/talos/talos/tests/'''tp5n'''`.

* see also [[#tp5|tp5 test]].

the tp6 pageset archives are also stored in their raw html (non-mitmdump archive) format on [https://github.com/rwood-moz/talos-pagesets github here]. If you wish to debug with the pagesets outside of mitmproxy then just clone that repo and you'll find them in the /talos-pagesets/tp6 folder.

'''Please keep these in alphabetical order'''

* contact: :surkov

* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/a11y a11y.manifest]

|}

* contact: :jaws

* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/about-preferences/about_preferences_basic.manifest]

|}

* contact: :jandem

* source: [https://searchfox.org/mozilla-central/source/third_party/webkit/PerformanceTests/ARES-6 ARES-6]

* unit: geometric mean / benchmark score

* contact: :davidb

* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/video video]

|}

* contact: :bholley

* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/perf-reftest perf-reftest]

|}

* contact: :bholley

* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/perf-reftest-singletons perf-reftest-singletons]

|}

* contact: :mconley

* measuring: Time from opening a new tab (which creates a new content process) to having that new content process be ready to load URLs.

|}

* contact: :ochameau

* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/devtools damp]

|}

* contact: :mattwoodrow

* source: [[https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/layout/benchmarks/displaylist_mutate.html displaylist_mutate.html]]

This measures the amount of time it takes to render a page after changing its display list. The page has a large number of display list items (10,000), and mutates one every frame. The goal of the test is to make displaylist construction a bottleneck, rather than painting or other factors, and thus improvements or regressions to displaylist construction will be visible. The test runs in ASAP mode to maximize framerate, and the result is how quickly the test was able to mutate and re-paint 600 items, one during each frame.

Dromaeo suite of tests for JavaScript performance testing. See the [[Dromaeo|Dromaeo wiki]] for more information.

Each sub-suite is divided into tests, and each test is divided into sub-tests.  Each sub-test takes some (in theory) fixed piece of work and measures how many times that piece of work can be performed in one second.  The score for a test is then the geometric mean of the runs/second numbers for its sub-tests.  The score for a sub-suite is the geometric mean of the scores for its tests.

* contact: :bz

* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/dromaeo css.manifest]

|}

* contact: :bz

* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/dromaeo dom.manifest]

Each page in the manifest is part of the dromaeo dom benchmark.  These are the specific areas that Dromaeo DOM covers:

Measures performance of getting and setting a DOM attribute, both via <code>getAttribute</code> and via a reflecting DOM property.  Also throws in some expando getting/setting for good measure.

Measures performance of various things that modify the DOM tree: creating element and text nodes and inserting them into the DOM.

Measures performance of various methods of looking for nodes in the DOM: <code>getElementById</code>, <code>getElementsByTagName</code>, and so forth.

Measures performance of various accessors (<code>childNodes</code>, <code>firstChild</code>, etc) that would be used when doing a walk over the DOM tree.

* contact: :jgilbert

* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/webgl/benchmarks/terrain glterrain]

|}

* contact: :jgilbert

* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/webgl/benchmarks/video glvideo]

This test playbacks a video file and ask WebGL to draw video frames as WebGL textures for 100 ticks. It collects the mean tick time across 100 ticks to measure how much time it will spend for a video texture upload to be a WebGL texture (gl.texImage2D). We run it for 5 times and ignore the first found. Lower results are better.

* contact: :jandem

* source: [[https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/jetstream/jetstream.manifest jetstream.manifest]] and jetstream.zip from tooltool

This is the [http://browserbench.org/JetStream/in-depth.html JetStream] javascript benchmark taken verbatim and slightly modified to fit into our pageloader extension and talos harness.

* contact: :sdetar

* source: [[https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/kraken kraken.manifest]]

|}

* contact: :davidb

* source: [[https://searchfox.org/mozilla-central/source/third_party/webkit/PerformanceTests/MotionMark source]] [[https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/motionmark manifests]]

** suite: we take a geometric mean of all the subtests (9 for animometer, 11 for html suite)

* contact: :bdahl

* source:  

|}

* contact: :rhunt

* source: [https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/gfx/benchmarks/rasterflood_svg.html rasterflood_svg.html]

Improvements (or regressions) to general painting performance or SVG are likely to affect this benchmark.

* contact: :rhunt

* source: [https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/gfx/benchmarks/rasterflood_gradient.html rasterflood_gradient.html]

The test runs for 10 seconds, and the resulting score is how many frames we were able to render during that time. Higher is better. Improvements (or regressions) to general painting performance or gradient rendering will affect this benchmark.

* contact: :mikedeboer, :mconley, :felipe

* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/startup_test/sessionrestore talos/sessionrestore]

|}

* contact: :selena

* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/speedometer/speedometer.manifest speedometer.manifest]

|}

* contact: :emilio

* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/stylebench/stylebench.manifest stylebench.manifest]

* reporting: runs/minute score

* contact: :mconley

* source: [[https://hg.mozilla.org/mozilla-central/file/tip/testing/talos/talos/startup_test/startup_about_home_paint/addon/]]

|}

* contact: :mconley

* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/tabpaint tabpaint]

** suite: geometric_mean(subtests)

* contact: :mconley

* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/tart tart]

|}

* contact: :jmaher

* source: [[#page_sets|tp5n.zip]]

|}

* '''nonmain_startup_fileio opt (with or without e10s) windows7-32''' – {{bug|1274018}} This test seems to consistently report a higher result for mozilla-central compared to Try even for an identical revision due to extension signing checks. In other words, if you are comparing Try and Mozilla-Central you may see a false-positive regression on perfherder. Graphs: [https://treeherder.mozilla.org/perf.html#/graphs?timerange=604800&series=%5Bmozilla-central,e5f5eaa174ef22fdd6b6e150e8c450aa827c2ff6,1,1%5D&series=%5Btry,e5f5eaa174ef22fdd6b6e150e8c450aa827c2ff6,1,1%5D non-e10s] [https://treeherder.mozilla.org/perf.html#/graphs?series=%5B%22mozilla-central%22,%222f3af3833d55ff371ecf01c41aeee1939ef3a782%22,1,1%5D&series=%5B%22try%22,%222f3af3833d55ff371ecf01c41aeee1939ef3a782%22,1,1%5D&timerange=604800 e10s]

A memory metric tracked during tp5 test runs. This metric is sampled every 20 seconds. This metric is collected on linux only.

[https://linux.die.net/man/3/xres xres man page].

Cpu usage tracked during tp5 test runs. This metric is sampled every 20 seconds. This metric is collected on windows only.

contact: :jimm, :overholt

</pre>

* contact: :kats

* source: [[#page_sets|tp5n.zip]]

|}

Some examples of things that cause regressions in this test are:

* Increased displayport size (which causes a larger display list to be built)

* Slowdown in rasterization of content

* contact: :rwood, :jmaher

* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/quantum_pageload tp6]

|}

Test pages used for tp6 are mitmproxy recordings that are played back during the tp6 test (and ultimately loaded in Firefox via the local proxy). Each test page is a separate mitmproxy recording (*.mp) file, however all recordings for the tp6 suite are archived in a single zip file on tooltool. When tp6 is run, talos automatically downloads the mitmproxy recording archive for use during the test.

* Select "No proxy" and click the "OK" button

* contact: :davidb

* source: [[https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/startup_test/tpaint.html tpaint-window.html]]

|}

* None listed yet. If you fix a regression for this test and have some tips to share, this is a good place for them.

* contact: :bdahl, :jimm, :jmaher

* source: [[https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/twinopen twinopen]]

|}

* contact: :mconley

* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/tabswitch tabswitch]

|}

* contact: :jimm

* source: [[https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/startup_test/tresize/addon/content/tresize-test.html tresize-test.html]]

|}

* slowdown in the paint pipeline

* resizes also trigger a rendering flush so bugs in the flushing code can manifest as regressions

* introduction of more spurious MozAfterPaint events - see {{bug|1471961}}

* contact: :davidb

* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/startup_test/tspaint_test.html tspaint_test.html]

|}

* (and/or maybe tpaint?) will regress if a new <panel> element is added to the browser window (e.g. browser.xul) and it's frame gets created. Fix this by ensuring it's display:none by default.

* contact: :jrmuizel

* source: [[https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/scroll scroll.manifest]]

|[https://groups.google.com/d/topic/mozilla.dev.platform/RICw5SJhNMo/discussion Replacing tscroll,tsvg with tscrollx,tsvgx]

|}

This test scrolls several pages where each represent a different known "hard" case to scroll (* needinfo), and measures the average frames interval (1/FPS) on each. The ASAP test (tscrollx) iterates in unlimited frame-rate mode thus reflecting the maximum scroll throughput per page.  To turn on ASAP mode, we set these preferences:

|}

* contact: :jwatt, :dholbert

* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/svg_opacity/svg_opacity.manifest]

|}

* contact: :jwatt, :dholbert, :neerja

* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/svg_static/ svg_static]

|}

* contact: :jwatt, :dholbert

* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/svgx svgx]

|}

* Did you change the dimensions of the content area? Even a little? The tsvgx test seems to be sensitive to changes like this. See {{bug|1375479}}, for example. Usually, these sorts of "regressions" aren't real regressions - they just mean that we need to re-baseline our expectations from the test.

* contact: :aklotz, :jmaher

* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/xtalos xperf instrumentation]

* [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/xperf.config#l11 'FileRead', 'FileWrite', 'FileFlush']

These are not part of the Talos code, but like Talos they are benchmarks that record data using the graphserver and are analyzed by the same scripts for regressions.

This test runs at build time and measures the number of static initializers in the compiled code.  Reducing this number is helpful for [https://blog.mozilla.org/tglek/2010/05/27/startup-backward-constructors/ startup optimizations].

** these are run for linux 32+64 opt and pgo builds.

* contact: :hsivonen

* source: [https://dxr.mozilla.org/mozilla-central/source/xpcom/tests/gtest/TestStrings.cpp]

Test whose name starts with PerfIsUTF8 test the performance of the XPCOM string IsUTF8 function with ASCII inputs if different lengths.

* The --enable-rust-simd accidentally getting turned off in automation.

* Changes to encoding_rs internals.

* LLVM optimizations regressing between updates to the copy of LLVM included in the Rust compiler.

* contact: :bholley

* source: [[https://dxr.mozilla.org/mozilla-central/source/testing/gtest/mozilla/MozGTestBench.cpp MozGTestBench.cpp]]

* summarization: Not a Talos test. This suite is provides a way to add low level platform performance regression tests for things that are not suited to be tested by Talos. See the [[[TestEngineering/Performance/Talos/Sheriffing#Microbench_Policy|Microbench Sheriffing Policy]]] for some notes on how to treat regressions.

* contact: :davidb

* source: https://dxr.mozilla.org/mozilla-central/source/xpcom/tests/gtest/TestStrings.cpp

PerfStripCharsCRLF() - call StripChars("\r\n") on 5 different test cases 20k times (each)

* contact: :bholley, :SimonSapin

* source: [https://dxr.mozilla.org/mozilla-central/source/layout/style/test/gtest]

* data: each test is run and measured 5 times

* summarization: take the [[TestEngineering/Performance/Talos/Data#median|median]] of the 5 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/gtest/mozilla/MozGTestBench.cpp#43-46 source: MozGTestBench.cpp]

Servo_StyleSheet_FromUTF8Bytes_Bench parses a sample stylesheet 20 times with Stylo’s CSS parser that is written in Rust. It starts from an in-memory UTF-8 string, so that I/O or UTF-16-to-UTF-8 conversion is not measured.