Auto-tools/Projects/Signal From Noise/Meetings/2012-03-15: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Created page with "= Previous Action Items = = Status Reports = == RelEng == * we now post to graphs.mozilla.org instead of the old one == Metrics == * Graphs to compare current method to propose...")
 
 
(9 intermediate revisions by 2 users not shown)
Line 1: Line 1:
= Previous Action Items =
= Previous Action Items =
* prove why 30 cycles is required - need to come up with a noise metric
* [[http://elvis314.wordpress.com/2012/03/12/reducing-the-noise-in-talos/ blog post]] outlining row major testing
* [http://people.mozilla.org/~ctalbert/TalosPlots/rowmajor_change/index.html Christina's plots]


= Status Reports =
= Status Reports =
== RelEng ==
== RelEng ==
* we now post to graphs.mozilla.org instead of the old one


== Metrics ==
== Metrics ==
* Graphs to compare current method to proposed method.


== Talos Changes ==
== Talos Changes ==
* tp5[10:30] - running 30 cycles of row major tp5, ignoring the first 10
* tp5[10:30] - running 30 cycles of row major tp5, ignoring the first 10
** [[http://people.mozilla.org/~jmaher/sxs/drop10/drop10.html Comparison table]] shows we are fluctuating less, still need more data points to prove it
** [[http://people.mozilla.org/~jmaher/sxs/drop10/drop10.html Comparison table]] shows we are fluctuating less, still need more data points to prove it
** runtimes are 38-42 minutes for 30 cycles, vs 16-19 for 10 cycles. Can we afford this?
** please look at the graphs on this, we observe a fix that landed for Firefox and our new numbers trend down with it, sometimes even better.
* {{bug|731391}} landed ignore first 10 data points
* {{bug|723571}} - pageloader shouldn't calculate statistics - DONE
** need to allow this to be configurable per test vs entire test file
* talos slaves can send data to s4n4 database
* {{bug|723571}} - working on cleaning up pageloader to not auto calculate statistics
** should we land this patch to have all slaves post?
* have raw data values going to s4n4 database, getting hole poked in IT firewall so we can do this from Pine/Try, etc..
 
 
 
* {{bug|727711}} - interpolate files referencing the talos directory with the ${talos} pattern. This will allow running of run_tests.py from outside of the talos directory.


== Graph Server ==
== Graph Server ==
*


== UI Implementation ==
== UI Implementation ==
* Added control panel that allows for selecting a collection of predefined firefox branches, talos tests, and platforms.  Also built in the same "advanced options" that the current graphs website provides.
Completed the talos test chart grid to visualize the mean, std deviation, and threshold for all branch/test/platform combinations for all test runs.
* Added a data view that provides average, min, max, standard deviation, and population variance for a given test runs values.
* Added generic visualization toggling.  Working on "Talos Regression Hunter" thumbnail visualization in d3.js


= Round Table =
= Round Table =
*
* steps required to get tp5[10:30] live
** post to dev.platform with announcement of change from column major->row major
** next week add new tp5 test names and changes to talos
** deploy talos and buildbot changes to turn on for side by side staging.
** need to define success criteria?
 
* can we visit tdhtml and tsvg next?


= Action Items =
= Action Items =
* prove why 30 cycles is required
* file bug to look into bi-model
* blog post outlining row major testing
* [[http://groups.google.com/group/mozilla.dev.platform/browse_thread/thread/f6691a23bf029c39# post to dev.platforms]] for column vs row
* [http://people.mozilla.org/~ctalbert/TalosPlots/rowmajor_change/index.html Christina's plots]
* decide on drop 5 run 25 (drop 5 keep 20, this runs in 26-29 minutes)
* gather a sample group of developers who can give feedback on new graph server ui
* send raw values from pine
* define a variance (noise) measurement

Latest revision as of 19:29, 16 March 2012

Previous Action Items

Status Reports

RelEng

Metrics

Talos Changes

  • tp5[10:30] - running 30 cycles of row major tp5, ignoring the first 10
    • [Comparison table] shows we are fluctuating less, still need more data points to prove it
    • please look at the graphs on this, we observe a fix that landed for Firefox and our new numbers trend down with it, sometimes even better.
  • bug 723571 - pageloader shouldn't calculate statistics - DONE
  • talos slaves can send data to s4n4 database
    • should we land this patch to have all slaves post?

Graph Server

UI Implementation

Completed the talos test chart grid to visualize the mean, std deviation, and threshold for all branch/test/platform combinations for all test runs.

Round Table

  • steps required to get tp5[10:30] live
    • post to dev.platform with announcement of change from column major->row major
    • next week add new tp5 test names and changes to talos
    • deploy talos and buildbot changes to turn on for side by side staging.
    • need to define success criteria?
  • can we visit tdhtml and tsvg next?

Action Items

  • file bug to look into bi-model
  • [post to dev.platforms] for column vs row
  • decide on drop 5 run 25 (drop 5 keep 20, this runs in 26-29 minutes)
  • gather a sample group of developers who can give feedback on new graph server ui
  • send raw values from pine
  • define a variance (noise) measurement
Retrieved from "https://wiki.allizom.org/index.php?title=Auto-tools/Projects/Signal_From_Noise/Meetings/2012-03-15&oldid=408610"