160
edits
TestEngineering/Performance/Sheriffing/Alert FAQ: Difference between revisions
TestEngineering/Performance/Sheriffing/Alert FAQ (view source)
Revision as of 12:11, 16 January 2018
, 16 January 2018expand a bit
(Perfherder now manages multiple types of alerts) |
(expand a bit) |
||
| Line 5: | Line 5: | ||
= Do we care about all alerts/tests = | = Do we care about all alerts/tests = | ||
Yes we do. Some tests are more commonly invalid, mostly due to the noise in the tests. We also adjust the threshold per test, the default is 2%, but for | Yes we do. Some tests are more commonly invalid, mostly due to the noise in the tests. We also adjust the threshold per test, the default is 2%, but for Dromaeo it is 5%. | ||
If we consider a test too noisy, we consider removing it entirely. | |||
Here are some platforms/tests which are exceptions about what we run: | Here are some platforms/tests which are exceptions about what we run: | ||
* | * Linux 64bit - the only platform which we run dromaeo_dom | ||
* | * Linux 32/64bit - the only platform in which no platform_microbench test runs, due to high noise levels | ||
* Windows 7 - the only platform that supports xperf (toolchain is only installed there) | * Windows 7 - the only platform that supports xperf (toolchain is only installed there) | ||
* Windows 7/10 - heavy profiles don't run here, because they take too long | |||
Lastly, we should prioritize alerts on the Mozilla-Beta | Lastly, we should prioritize alerts on the Mozilla-Beta branch since those are affecting more people. | ||
= What does a regression look like on the graph = | = What does a regression look like on the graph = | ||