View source for Buildbot/Talos/Data

= Overview =

Historically, Talos has collected data in the Graph Server.  With the focus on noise in the Talos results, the Signal From Noise project was created and one of the resulting projects is Datazilla.  Datazilla is intended to replace graph server in due time, right now it is in the beta testing stage.

= Graph Server (Current solution) =

== How are the numbers calculated? ==

To ensure that the base profile is correctly installed for every test the browser is opened/closed once before test execution.  This first cold open is excluded for the test result calculation.  

For "cold" tests the caches are cleared after this initial open/close - in this way the browser is configured and ready but returned to a "cold" state.

All tests are run with newly installed profiles - profiles are not shared across test runs.

=== Pageload style tests (tp5, tdhtml, etc) ===

The overall test number is determined by first calculating the median page load time for each page in the set (excluding the max page load per individual page). The max median from that set is then excluded and the average is taken; that becomes the number reported to the tinderbox waterfall.

=== Ts style tests (ts, twinopen, ts_cold, etc) ===

The overall test number is calculated by excluding the max opening time and taking an average of the remaining numbers.

== Where are the numbers stored? ==
The results of every talos test are reported to the [https://wiki.mozilla.org/Perfomatic Perfomatic] graph server.  When running locally, you can specify output to a file using the --results_url argument to PerfConfigurator, e.g.

    PerfConfigurator --activeTests tsvg -e `which firefox` -o tsvg.yml --results_url file://${PWD}/tsvg.txt

[[Buildbot/Talos/DataFormat|Talos data formatting]]

== Regressions ==

To determine whether a good point is "good" or "bad", we take 12 points of historical data, and 12 points of new data. 
We compare these using a [http://en.wikipedia.org/wiki/Student%27s_t-test t-test].  See https://wiki.mozilla.org/images/c/c0/Larres-thesis.pdf#page=74 .

Regressions are mailed to the [https://lists.mozilla.org/listinfo/dev-tree-management dev-tree-management] mailing list.  Regressions are calculated by the [http://hg.mozilla.org/graphs/file/tip/server/analysis/analyze_talos.py analyze_talos.py] script which uses a configuration file based on http://hg.mozilla.org/graphs/file/tip/server/analysis/analysis.cfg.template

= Datazilla (future solution) =

Datazilla is collecting data from all our talos runs just like graph server.  The difference is that it is collecting the raw data instead of pre calculated data about the tests.