TestEngineering/Performance/Raptor: Difference between revisions

TestEngineering/Performance/Raptor (view source)

765 bytes added , 11 November 2020

Add high-value test information.

83

edits

@@ Line 87: / Line 87: @@
 When it's removed there will no longer be a <tt>shouldAlert</tt> field in the output perferhder data (you can find the [https://searchfox.org/mozilla-central/source/testing/mozharness/external_tools/performance-artifact-schema.json#68,165 schema here]). As long as <tt>shouldAlert</tt> is not in the data, no alerts will be generated. If you need to also disable code sheriffing for the test, then you need to change the tier of the task to 3.
+==== High value tests ====
+We have a notion of '''high-value''' tests in performance testing. These are chosen based on how many alerts they can catch using [https://github.com/gmierz/moz-current-tests/blob/master/high-value-tests/generate_high_value_tests.py this script] along with [https://github.com/gmierz/moz-current-tests/blob/master/high-value-tests/sql_query.txt a redash query]. The lists below are the minimum set of test pages we should run to catch as many alerts as we can.
+'''Desktop'''
+Last updated: June 2020
+    amazon
+    gmail
+    gslides
+    imgur
+    tumblr
+    twitch
+    twitter
+'''Mobile'''
+Last updated: November 2020
+    allrecipes
+    amazon-search
+    espn
+    facebook
+    google-search
+    microsoft-support
+    youtube-watch
 === Benchmark Tests ===