947
edits
Auto-tools/Projects/Signal From Noise/Execution2012: Difference between revisions
Jump to navigation
Jump to search
Auto-tools/Projects/Signal From Noise/Execution2012 (view source)
Revision as of 23:03, 28 December 2012
, 28 December 2012→Goals of Signal from Noise: reword
(→Goals of Signal from Noise: reword) |
(→Goals of Signal from Noise: reword) |
||
| Line 8: | Line 8: | ||
** Document and expose these statistical methodologies in such a way that they are transparent | ** Document and expose these statistical methodologies in such a way that they are transparent | ||
* Make it easier/possible for developers to detect regressions in their code both locally and from [https://wiki.mozilla.org/Build:TryServer try server]. | * Make it easier/possible for developers to detect regressions in their code both locally and from [https://wiki.mozilla.org/Build:TryServer try server]. | ||
* Turn talos jobs on TBPL orange when a talos regression is detected. Developers look at https://tbpl.mozilla.org/ as the singular source of truth for what is good or bad for a given push. | * Turn talos jobs on TBPL orange when a talos regression is detected. Developers look at https://tbpl.mozilla.org/ as the singular source of truth for what is good or bad for a given push. | ||
It is also an implicit goal of Signal from Noise to ensure that we are measuring as close as possible to user-relevent statistics. Performance tests are | |||
It is also an implicit goal of Signal from Noise to ensure that we are measuring as close as possible to user-relevent statistics. Performance tests are effectively a proxy for what the user experiences; while it is only an analog, a principle of performance testing is that you measure analogs that have some meaning to the user. While e.g. Larres' proposal to turn off address space randomization to eliminates noise, it also creates a system under test that is much different from how the user would experience the system. This also makes it harder, in general, to test, since each developer would have to build a non-ASR version of Firefox for Talos testing. | |||
== Execution of Signal from Noise == | == Execution of Signal from Noise == | ||