TestEngineering/Performance/Glossary: Difference between revisions

TestEngineering/Performance/Glossary (view source)

559 bytes added , 6 January 2020

342

edits

@@ Line 32: / Line 32: @@
 == Noise (regarding performance tests) ==
-Definition needed
+Generally a test reports values that are in a range instead of a consistent value. The larger the range of 'normal' results, the more noise we have.
+Some tests will post results in a small range, and when we get a data point significantly outside the range, it is easy to identify.
+The problem is that many tests have a large range of expected results. It makes it hard to determine what a regression is when we might have a range += 4% from the median and we have a 3% regression. It is obvious in the graph over time, but hard to tell until you have many future data points.
 == Nudge ==