342
edits
TestEngineering/Performance/FAQ: Difference between revisions
TestEngineering/Performance/FAQ (view source)
Revision as of 07:55, 29 January 2020
, 29 January 2020→What is Noise
Alexandrui (talk | contribs) |
Alexandrui (talk | contribs) |
||
| Line 187: | Line 187: | ||
Some tests will post results in a small range, and when we get a data point significantly outside the range, it is easy to identify. | Some tests will post results in a small range, and when we get a data point significantly outside the range, it is easy to identify. | ||
The problem is that many tests have a large range of expected results. It makes it hard to determine what a regression is when we might have a range += 4% from the median and we have a 3% regression. It is obvious in the graph over time, but hard to tell until you have many future data points. | The problem is that many tests have a large range of expected results (we call them unstable). It makes it hard to determine what a regression is when we might have a range += 4% from the median and we have a 3% regression. It is obvious in the graph over time, but hard to tell until you have many future data points. | ||
<br /> | |||
[[File:Noisy graph.png|Noisy graph]] | |||
=== Why can we not trust a single data point === | === Why can we not trust a single data point === | ||