Confirmed users
372
edits
TestEngineering/Performance/Talos/Tests: Difference between revisions
Jump to navigation
Jump to search
TestEngineering/Performance/Talos/Tests (view source)
Revision as of 17:47, 22 January 2018
, 22 January 2018→perf-reftest update
| Line 180: | Line 180: | ||
** suite: identical to subtest | ** suite: identical to subtest | ||
The perf-reftest suite is a unique talos suite where each subtest loads two different test pages: a 'base' page (i.e. bloom_basic) and a 'reference' page (i.e. bloom_basic_ref), and then compares each of the page load times against eachother to determine the variance. | Style system performance test suite. The perf-reftest suite is a unique talos suite where each subtest loads two different test pages: a 'base' page (i.e. bloom_basic) and a 'reference' page (i.e. bloom_basic_ref), and then compares each of the page load times against eachother to determine the variance. | ||
Talos runs each of the two pages as if they are stand-alone tests, and then calculates and reports the variance; the test output 'replicates' reported from bloom_basic are actually the comparisons between the 'base' and 'reference' pages for each page load cycle. The suite contains multiple subtests, each of which contains a base page and a reference page. | Talos runs each of the two pages as if they are stand-alone tests, and then calculates and reports the variance; the test output 'replicates' reported from bloom_basic are actually the comparisons between the 'base' and 'reference' pages for each page load cycle. The suite contains multiple subtests, each of which contains a base page and a reference page. | ||
If you wish to see the individual 'base' and 'reference' page results instead of just the reported difference, the 'base_replicates' and 'ref_replicates' can be found in the PERFHERDER_DATA log file output, and in the 'local.json' talos output file when running talos locally. In production, both of the page replicates are also archived in the perfherder-data.json file. The perfherder-data.json file is archived after each run in production, and can be found on the Treeherder Job Details tab when the perf-reftest job symbol is selected. | If you wish to see the individual 'base' and 'reference' page results instead of just the reported difference, the 'base_replicates' and 'ref_replicates' can be found in the PERFHERDER_DATA log file output, and in the 'local.json' talos output file when running talos locally. In production, both of the page replicates are also archived in the perfherder-data.json file. The perfherder-data.json file is archived after each run in production, and can be found on the Treeherder Job Details tab when the perf-reftest job symbol is selected. | ||
This test suite was ported over from the style-perf-tests (https://github.com/heycam/style-perf-tests). | |||
{| class="mw-collapsible mw-collapsed wikitable" | {| class="mw-collapsible mw-collapsed wikitable" | ||