Auto-tools/Projects/Signal From Noise/StatusNovember2011: Difference between revisions

(from https://etherpad.mozilla.org/GxTsTCvpAo)
 
Line 3: Line 3:
== Review of Current Performance Testing Workflow ==
== Review of Current Performance Testing Workflow ==


For a push to (e.g.) mozilla-central:
For a push to (e.g.) [https://hg.mozilla.org/mozilla-central/ mozilla-central]:


* talos tests are run by buildbot on the test slaves
* [https://wiki.mozilla.org/Buildbot/Talos talos] tests are run by buildbot on the test slaves
* talos uploads numbers to graphs.mozilla.org from the slaves
* talos uploads numbers to http://graphs.mozilla.org from the slaves
* a script versioned with graphserver (but unrelated to and unused by graphs.mozilla.org) mails dev-tree-management when a regression or improvement is detected: http://hg.mozilla.org/graphs/file/tip/server/analysis/analyze_talos.py
* a script versioned with graphserver (but unrelated to and unused by graphs.mozilla.org) mails dev-tree-management when a regression or improvement is detected: http://hg.mozilla.org/graphs/file/tip/server/analysis/analyze_talos.py


It is a common misconception that TBPL talos tests turned orange for test failure.  In fact, as long as no infrastructure problems are encountered, Talos runs always report green regardless of the performance of the system under test.  Graphserver and the talos regression emails are the only source of information as to whether a regression or improvement has resulted.  However, the existing methods of calculating statistics (professing to detect regressions as low as 1%)  lead to a high degree of noise.  The consequence is confusion amongst developers with regard to whether a reported regression is legitimate or not; the consequence of this is that the regression emails are largely ignored since there are so many of them and most of them are merely noise.
It is a common misconception that [https://tbpl.mozilla.org/ TBPL] talos tests turned orange for test failure.  In fact, as long as no infrastructure problems are encountered, Talos runs always report green regardless of the performance of the system under test.  Graphserver and the talos regression emails are the only source of information as to whether a regression or improvement has resulted.  However, the existing methods of calculating statistics (professing to detect regressions as low as 1%)  lead to a high degree of noise.  The consequence is confusion amongst developers with regard to whether a reported regression is legitimate or not; the consequence of this is that the regression emails are largely ignored since there are so many of them and most of them are merely noise.


It is another common misconception that all of the "Talos regression/improvement" emails actually come from Talos runs.  Some (e.g. numbers of constructors) do not.
It is another common misconception that all of the "Talos regression/improvement" emails actually come from Talos runs.  Some (e.g. numbers of constructors) do not.
947

edits