MozillaWiki

TestEngineering/Performance/Talos/Tests: Difference between revisions

Page Discussion

TestEngineering/Performance/Talos/Tests (view source)

Revision as of 16:49, 22 January 2018

34 bytes added ,  22 January 2018
reorganized
(removed possible regression causes to clean up page)
(reorganized)
Line 83: Line 83:
* see also [[#tp5|tp5 test]].
* see also [[#tp5|tp5 test]].


== Test Descriptions ==
== Talos Test Descriptions ==


'''Please keep these in alphabetical order'''
'''Please keep these in alphabetical order'''
Line 438: Line 438:


This test playbacks a video file and ask WebGL to draw video frames as WebGL textures for 100 ticks. It collects the mean tick time across 100 ticks to measure how much time it will spend for a video texture upload to be a WebGL texture (gl.texImage2D). We run it for 5 times and ignore the first found. Lower results are better.
This test playbacks a video file and ask WebGL to draw video frames as WebGL textures for 100 ticks. It collects the mean tick time across 100 ticks to measure how much time it will spend for a video texture upload to be a WebGL texture (gl.texImage2D). We run it for 5 times and ignore the first found. Lower results are better.
=== IsASCII and IsUTF8 gtest microbenchmarks ===
* contact: :hsivonen
* source: [https://dxr.mozilla.org/mozilla-central/source/xpcom/tests/gtest/TestStrings.cpp]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#Microbench Microbench]
* reporting: intervals in ms (lower is better)
* data: each test is run and measured 5 times
* summarization: take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the 5 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/gtest/mozilla/MozGTestBench.cpp#43-46 source: MozGTestBench.cpp]
Test whose name starts with PerfIsASCII test the performance of the XPCOM string IsASCII function with ASCII inputs if different lengths.
Test whose name starts with PerfIsUTF8 test the performance of the XPCOM string IsUTF8 function with ASCII inputs if different lengths.
==== Possible regression causes ====
* The --enable-rust-simd accidentally getting turned off in automation.
* Changes to encoding_rs internals.
* LLVM optimizations regressing between updates to the copy of LLVM included in the Rust compiler.


=== kraken ===
=== kraken ===
Line 490: Line 472:
</pre>
</pre>
|}
|}
=== Microbench ===
* contact:
* source: [[https://dxr.mozilla.org/mozilla-central/source/testing/gtest/mozilla/MozGTestBench.cpp MozGTestBench.cpp]]
* type: Custom GTest micro-benchmarking
* data: Time taken for a GTest function to execute
* summarization: Not a Talos test. This suite is provides a way to add low level platform performance regression tests for things that are not suited to be tested by Talos. See the [[https://wiki.mozilla.org/Performance_sheriffing/Talos/Sheriffing#Microbench_Policy Microbench Sheriffing Policy]] for some notes on how to treat regressions.
=== Other data ===
These are not part of the Talos code, but like Talos they are benchmarks that record data using the graphserver and are analyzed by the same scripts for regressions.
==== Number of Constructors (num_ctors) ====
This test runs at build time and measures the number of static initializers in the compiled code.  Reducing this number is helpful for [https://blog.mozilla.org/tglek/2010/05/27/startup-backward-constructors/ startup optimizations].
* https://hg.mozilla.org/build/tools/file/348853aee492/buildfarm/utils/count_ctors.py
** these are run for linux 32+64 opt and pgo builds.
=== PerfStrip Tests ===
* contact: :milan
* source: https://dxr.mozilla.org/mozilla-central/source/xpcom/tests/gtest/TestStrings.cpp
* type: Microbench
* reporting: execution time in ms (lower is better) for 100k function calls
* data: each test run and measured 5 times
* summarization:
PerfStripWhitespace - call StripWhitespace() on 5 different test cases 20k times (each)
PerfStripCharsWhitespace - call StripChars("\f\t\r\n") on 5 different test cases 20k times (each)
PerfStripCRLF - call StripCRLF() on 5 different test cases 20k times (each)
PerfStripCharsCRLF() - call StripChars("\r\n") on 5 different test cases 20k times (each)


=== rasterflood_svg ===
=== rasterflood_svg ===
Line 754: Line 700:
</pre>
</pre>
|}
|}
=== Stylo gtest microbenchmarks ===
* contact: :bholley, :SimonSapin
* source: [https://dxr.mozilla.org/mozilla-central/source/layout/style/test/gtest]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#Microbench Microbench]
* reporting: intervals in ms (lower is better)
* data: each test is run and measured 5 times
* summarization: take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the 5 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/gtest/mozilla/MozGTestBench.cpp#43-46 source: MozGTestBench.cpp]
Servo_StyleSheet_FromUTF8Bytes_Bench parses a sample stylesheet 20 times with Stylo’s CSS parser that is written in Rust. It starts from an in-memory UTF-8 string, so that I/O or UTF-16-to-UTF-8 conversion is not measured.
Gecko_nsCSSParser_ParseSheet_Bench does the same with Gecko’s previous CSS parser that is written in C++, for comparison.
Servo_DeclarationBlock_SetPropertyById_Bench parses the string "10px" with Stylo’s CSS parser and sets it as the value of a property in a declaration block, a million times. This is similar to animations that are based on JavaScript code modifying Element.style instead of using CSS @keyframes.
Servo_DeclarationBlock_SetPropertyById_WithInitialSpace_Bench is the same, but with the string " 10px" with an initial space. That initial space is less typical of JS animations, but is almost always there in stylesheets or full declarations like "width: 10px". This microbenchmark was used to test the effect of some specific code changes. Regressions here may be acceptable if Servo_StyleSheet_FromUTF8Bytes_Bench is not affected.


=== tabpaint ===
=== tabpaint ===
Line 799: Line 727:


=== TART ===
=== TART ===
* contact: :MattN, :mconley, :jmaher,
* contact: :MattN, :mconley  
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/tart tart]
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/tart tart]
* type: PageLoader
* type: PageLoader
Line 868: Line 796:
</pre>
</pre>
|}
|}
=== TestStandardURL ===
* contact: :valentin
* source:
* type:
* reporting:
* data:
* summarization:
TestStandardURL.Perf - Runs 10K iterations on the most important getters and setters of nsStandardURL
TestStandardURL.NormalizePerf - Runs 20K x 5 iterations on nsStandardURL::NormalizeIPv4 on input that are valid IPv4 strings
TestStandardURL.NormalizePerfFails - Runs 20K x 5 iterations on nsStandardURL::NormalizeIPv4 on input that is not valid IPv4


=== tp5 ===
=== tp5 ===
Line 1,470: Line 1,383:
The values we collect during stackwalk are:
The values we collect during stackwalk are:
* [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/xperf.config#l11 'FileRead', 'FileWrite', 'FileFlush']
* [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/xperf.config#l11 'FileRead', 'FileWrite', 'FileFlush']
== Build Metrics ==
These are not part of the Talos code, but like Talos they are benchmarks that record data using the graphserver and are analyzed by the same scripts for regressions.
=== Number of Constructors (num_ctors) ===
This test runs at build time and measures the number of static initializers in the compiled code.  Reducing this number is helpful for [https://blog.mozilla.org/tglek/2010/05/27/startup-backward-constructors/ startup optimizations].
* https://hg.mozilla.org/build/tools/file/348853aee492/buildfarm/utils/count_ctors.py
** these are run for linux 32+64 opt and pgo builds.
== Platform Microbenchmark Tests ==
=== IsASCII and IsUTF8 gtest microbenchmarks ===
* contact: :hsivonen
* source: [https://dxr.mozilla.org/mozilla-central/source/xpcom/tests/gtest/TestStrings.cpp]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#Microbench Microbench]
* reporting: intervals in ms (lower is better)
* data: each test is run and measured 5 times
* summarization: take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the 5 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/gtest/mozilla/MozGTestBench.cpp#43-46 source: MozGTestBench.cpp]
Test whose name starts with PerfIsASCII test the performance of the XPCOM string IsASCII function with ASCII inputs if different lengths.
Test whose name starts with PerfIsUTF8 test the performance of the XPCOM string IsUTF8 function with ASCII inputs if different lengths.
==== Possible regression causes ====
* The --enable-rust-simd accidentally getting turned off in automation.
* Changes to encoding_rs internals.
* LLVM optimizations regressing between updates to the copy of LLVM included in the Rust compiler.
=== Microbench ===
* contact:
* source: [[https://dxr.mozilla.org/mozilla-central/source/testing/gtest/mozilla/MozGTestBench.cpp MozGTestBench.cpp]]
* type: Custom GTest micro-benchmarking
* data: Time taken for a GTest function to execute
* summarization: Not a Talos test. This suite is provides a way to add low level platform performance regression tests for things that are not suited to be tested by Talos. See the [[https://wiki.mozilla.org/Performance_sheriffing/Talos/Sheriffing#Microbench_Policy Microbench Sheriffing Policy]] for some notes on how to treat regressions.
=== PerfStrip Tests ===
* contact: :milan
* source: https://dxr.mozilla.org/mozilla-central/source/xpcom/tests/gtest/TestStrings.cpp
* type: Microbench
* reporting: execution time in ms (lower is better) for 100k function calls
* data: each test run and measured 5 times
* summarization:
PerfStripWhitespace - call StripWhitespace() on 5 different test cases 20k times (each)
PerfStripCharsWhitespace - call StripChars("\f\t\r\n") on 5 different test cases 20k times (each)
PerfStripCRLF - call StripCRLF() on 5 different test cases 20k times (each)
PerfStripCharsCRLF() - call StripChars("\r\n") on 5 different test cases 20k times (each)
=== Stylo gtest microbenchmarks ===
* contact: :bholley, :SimonSapin
* source: [https://dxr.mozilla.org/mozilla-central/source/layout/style/test/gtest]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#Microbench Microbench]
* reporting: intervals in ms (lower is better)
* data: each test is run and measured 5 times
* summarization: take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the 5 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/gtest/mozilla/MozGTestBench.cpp#43-46 source: MozGTestBench.cpp]
Servo_StyleSheet_FromUTF8Bytes_Bench parses a sample stylesheet 20 times with Stylo’s CSS parser that is written in Rust. It starts from an in-memory UTF-8 string, so that I/O or UTF-16-to-UTF-8 conversion is not measured.
Gecko_nsCSSParser_ParseSheet_Bench does the same with Gecko’s previous CSS parser that is written in C++, for comparison.
Servo_DeclarationBlock_SetPropertyById_Bench parses the string "10px" with Stylo’s CSS parser and sets it as the value of a property in a declaration block, a million times. This is similar to animations that are based on JavaScript code modifying Element.style instead of using CSS @keyframes.
Servo_DeclarationBlock_SetPropertyById_WithInitialSpace_Bench is the same, but with the string " 10px" with an initial space. That initial space is less typical of JS animations, but is almost always there in stylesheets or full declarations like "width: 10px". This microbenchmark was used to test the effect of some specific code changes. Regressions here may be acceptable if Servo_StyleSheet_FromUTF8Bytes_Bench is not affected.
=== TestStandardURL ===
* contact: :valentin
* source:
* type:
* reporting:
* data:
* summarization:
TestStandardURL.Perf - Runs 10K iterations on the most important getters and setters of nsStandardURL
TestStandardURL.NormalizePerf - Runs 20K x 5 iterations on nsStandardURL::NormalizeIPv4 on input that are valid IPv4 strings
TestStandardURL.NormalizePerfFails - Runs 20K x 5 iterations on nsStandardURL::NormalizeIPv4 on input that is not valid IPv4


* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests duplicated & updated from old page]
* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests duplicated & updated from old page]
Jmaher
Confirmed users
3,376

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/1187265"