Auto-tools/Projects/Signal From Noise/Execution2012: Difference between revisions

Auto-tools/Projects/Signal From Noise/Execution2012 (view source)

Revision as of 23:10, 28 December 2012

61 bytes added , 28 December 2012

→‎Goals of Signal from Noise: link to where Larres proposes turning off ASR

Jhammel

947

edits

@@ Line 10: / Line 10: @@
 * Turn talos jobs on TBPL orange when a talos regression is detected. Developers look at https://tbpl.mozilla.org/ as the singular source of truth for what is good or bad for a given push.
-It is also an implicit goal of Signal from Noise to ensure that we are measuring as close as possible to user-relevent statistics. Performance tests are effectively a proxy for what the user experiences; while the test is only an analog, a principle of performance testing is that you measure analogs that have some meaning to the user.  While e.g. Larres' proposal to turn off address space randomization to eliminates noise, it also creates a system under test that is much different from how the user would experience the system.  This also makes it harder, in general, to test, since each developer would have to build a non-ASR version of Firefox for Talos testing.
+It is also an implicit goal of Signal from Noise to ensure that we are measuring as close as possible to user-relevant statistics. Performance tests are effectively a proxy for what the user experiences; while the test is only an analog, a principle of performance testing is that you measure analogs that have some meaning to the user.  While e.g. Larres' proposal to turn off address space randomization ( https://wiki.mozilla.org/images/c/c0/Larres-thesis.pdf#page=55 ) eliminates noise, it also creates a system under test that is much different from how the user would experience Firefox.  This also makes it harder, in general, to test, since each developer would have to build a non-ASR version of Firefox for Talos testing.
 == Execution of Signal from Noise ==