342
edits
TestEngineering/Performance/FAQ: Difference between revisions
TestEngineering/Performance/FAQ (view source)
Revision as of 08:39, 29 January 2020
, 29 January 2020→Noise
Alexandrui (talk | contribs) |
Alexandrui (talk | contribs) (→Noise) |
||
| Line 191: | Line 191: | ||
[[File:Noisy graph.png|Noisy graph]] | [[File:Noisy graph.png|Noisy graph]] | ||
=== What are low value tests? === | |||
In the context of noise, the low value mean that the regression magnitude is too small related to the noise of the tests, thus it's pretty hard to tell which particular bug/commit caused this, but rather a range. | |||
<br /> | |||
In a sheriffing perspective, those often end up as WONTFIX/INVALID or tests which are often considered unreliable, not relevant to current Firefox revision etc. | |||
<br /> | |||
[[File:Noisy low value graph.png.png|Noisy low value graph]] | |||
=== Why can we not trust a single data point === | === Why can we not trust a single data point === | ||
This is a problem we have dealt with for years with no perfect answer. Some reasons we do know are: | This is a problem we have dealt with for years with no perfect answer. Some reasons we do know are: | ||