TestEngineering/Performance/Raptor: Difference between revisions

Add high-value test information.
(Add high-value test information.)
Line 87: Line 87:


When it's removed there will no longer be a <tt>shouldAlert</tt> field in the output perferhder data (you can find the [https://searchfox.org/mozilla-central/source/testing/mozharness/external_tools/performance-artifact-schema.json#68,165 schema here]). As long as <tt>shouldAlert</tt> is not in the data, no alerts will be generated. If you need to also disable code sheriffing for the test, then you need to change the tier of the task to 3.
When it's removed there will no longer be a <tt>shouldAlert</tt> field in the output perferhder data (you can find the [https://searchfox.org/mozilla-central/source/testing/mozharness/external_tools/performance-artifact-schema.json#68,165 schema here]). As long as <tt>shouldAlert</tt> is not in the data, no alerts will be generated. If you need to also disable code sheriffing for the test, then you need to change the tier of the task to 3.
==== High value tests ====
We have a notion of '''high-value''' tests in performance testing. These are chosen based on how many alerts they can catch using [https://github.com/gmierz/moz-current-tests/blob/master/high-value-tests/generate_high_value_tests.py this script] along with [https://github.com/gmierz/moz-current-tests/blob/master/high-value-tests/sql_query.txt a redash query]. The lists below are the minimum set of test pages we should run to catch as many alerts as we can.
'''Desktop'''
Last updated: June 2020
    amazon
    gmail
    gslides
    imgur
    tumblr
    twitch
    twitter
'''Mobile'''
Last updated: November 2020
    allrecipes
    amazon-search
    espn
    facebook
    google-search
    microsoft-support
    youtube-watch


=== Benchmark Tests ===
=== Benchmark Tests ===
83

edits