947
edits
Auto-tools/Projects/Signal From Noise/Execution2012: Difference between revisions
Jump to navigation
Jump to search
Auto-tools/Projects/Signal From Noise/Execution2012 (view source)
Revision as of 23:56, 17 December 2012
, 17 December 2012→Goals of Signal from Noise: link
(→Signal From Noise: remove link that no longer exists) |
(→Goals of Signal from Noise: link) |
||
| Line 6: | Line 6: | ||
* Understand our performance statistics models and assumptions end-to-end and ensure that the statistics we are using are valid | * Understand our performance statistics models and assumptions end-to-end and ensure that the statistics we are using are valid | ||
** Document and expose these statistics in such a way that they are transparent | ** Document and expose these statistics in such a way that they are transparent | ||
* Make it easier/possible for developers to detect regressions in their code both locally and from try server. | * Make it easier/possible for developers to detect regressions in their code both locally and from [https://wiki.mozilla.org/Build:TryServer try server]. | ||
* Turn talos jobs on TBPL orange when a talos regression is detected. Developers look at https://tbpl.mozilla.org/ as the singular source of truth for what is good or bad for a given push. | * Turn talos jobs on TBPL orange when a talos regression is detected. Developers look at https://tbpl.mozilla.org/ as the singular source of truth for what is good or bad for a given push. | ||
It is also an implicit goal of Signal from Noise to ensure that we are measuring as close as possible to user-relevent statistics. Performance tests are essentially a proxy for what the user experiences; while it is only an analog, a principle of performance testing is that you measure analogs that have some meaning to the user. While e.g. Larres' proposal to turn off address space randomization to cut out the noise it introduces, it also creates a system under test that is much different from how the user would experience the system. This also makes it harder, in general, to test, since each developer would have to build a non-ASR version of Firefox for Talos testing. | It is also an implicit goal of Signal from Noise to ensure that we are measuring as close as possible to user-relevent statistics. Performance tests are essentially a proxy for what the user experiences; while it is only an analog, a principle of performance testing is that you measure analogs that have some meaning to the user. While e.g. Larres' proposal to turn off address space randomization to cut out the noise it introduces, it also creates a system under test that is much different from how the user would experience the system. This also makes it harder, in general, to test, since each developer would have to build a non-ASR version of Firefox for Talos testing. | ||
== Execution of Signal from Noise == | == Execution of Signal from Noise == | ||
In general, it was initially expected that some back-of-the-envelope analysis of existing Talos numbers and some (again) back-of-the-envelope engineering implementations of statistics that at least appeared (though would probably be non-rigorous) less noisy would satisfy the goals of the Signal from Noise project for the time being, with optimistically ongoing effort being invested in analysis of performance data. The initial SfN effort was scheduled for a single quarter. This proved rather optimistic. (See also: http://k0s.org/mozilla/blog/20120829151007 ) | In general, it was initially expected that some back-of-the-envelope analysis of existing Talos numbers and some (again) back-of-the-envelope engineering implementations of statistics that at least appeared (though would probably be non-rigorous) less noisy would satisfy the goals of the Signal from Noise project for the time being, with optimistically ongoing effort being invested in analysis of performance data. The initial SfN effort was scheduled for a single quarter. This proved rather optimistic. (See also: http://k0s.org/mozilla/blog/20120829151007 ) | ||