947
edits
Auto-tools/Projects/Signal From Noise/Execution2012: Difference between revisions
Jump to navigation
Jump to search
Auto-tools/Projects/Signal From Noise/Execution2012 (view source)
Revision as of 23:57, 28 December 2012
, 28 December 2012→Problems We Aimed to Solve with Datazilla
(→Problems We Aimed to Solve with Datazilla: link to data.sql + formatting) |
|||
| Line 24: | Line 24: | ||
* Preserve and capture raw performance numbers: the Talos test framework is a bad place to do statistics, because if you do any averaging before uploading the results then the ability to retrieve the original data is forever lost. Instead, datazilla should take in all raw values from talos and provide a central platform for regression/improvement detection and statistical study. | * Preserve and capture raw performance numbers: the Talos test framework is a bad place to do statistics, because if you do any averaging before uploading the results then the ability to retrieve the original data is forever lost. Instead, datazilla should take in all raw values from talos and provide a central platform for regression/improvement detection and statistical study. | ||
* Reduce the granularity of Talos from a page set to a single page: | * Reduce the granularity of Talos from a page set to a single page: statistics and regressions should be dealt with on a per-page basis, as pages may have wildly different performance characteristics. See also https://wiki.mozilla.org/Metrics/Talos_Investigation#Unrolling_Talos and http://k0s.org/mozilla/blog/20120425093346 . | ||
* Establish a full, extensible RESTful interface to the data: Datazilla's data and statistical methods should be accessible by all developers and the tools they wish to | * Establish a full, extensible RESTful interface to the data: Datazilla's data and statistical methods should be accessible by all developers and the tools they may wish to craft to use the data. | ||
* Statistics should be self-evident: often, Talos+Graphserver and other statistical systems have been approached as a "black box": A number comes out that is "good" or "bad". However, this effectively leaves an interested developer in the dark as to where this number came from and discourages understanding the system and playing with data. Datazilla was designed to expose the statistics being used so that there are no mysteries here. | * Statistics should be self-evident: often, Talos+Graphserver and other statistical systems have been approached as a "black box": A number comes out that is "good" or "bad". However, this effectively leaves an interested developer in the dark as to where this number came from and discourages understanding the system and playing with data. Datazilla was designed to expose the statistics being used so that there are no mysteries here. | ||