View source for Auto-tools/Projects/Signal From Noise/Meetings/2012-06-28

= Previous Action Items =
* ttest data: ensure we drop the first 5, find a bad changset, view the python code
* determine failed page threshold.  If we have 1 page regressed, do we call the push failed?
* [jmaher] talos branch names and operating systems to match tbpl, etc...

= Goals =
== a production system slurping up data from talos deployed ==
** Owner: Jeads
** Next steps: https://www.pivotaltracker.com/epic/show/107545

== Metrics for determining page-centric "did my test pass or not" for at least one suite dromaeo and ts_paint ==
** Owner: Christina/Jeads
** Next steps: Iterate on the development data.  We're going to go ahead with the ~26 "good" pages in tp5.  It's going to be awhile before we have representative data for dromaeo and wading in on ts_paint could introduce a whole new set of variables that send us into the talos briar patch.
*** [jeads] Build SQL queries to pull out representative data for the ~26 "good" pages in tp5
*** [christina and jeads] Develop the input data set for analysis in R
*** [christina and jeads] Identify an outlier that is outside an allowable range
*** [christina] Develop an allowable range for a test result for each of the ~26 pages
*** [christina] Determine a score for a changeset across the entire test suite.  This could be as simple as 0 or 1 for each page, that gets added up across all pages and platforms. So 24/26 would indicate 24 pages were in the allowable range 2 were not for one platform.  We will not know the best way to do this until we wade into the data.
*** [everyone] Determine a way to convert the test suite score into a pass, fail, or undetermined result for a given changeset. 

== a stage tbpl system mocking the Yes/No/Maybe interaction ==
** Owner: jmaher/jhammel - eta: after we are side by side staging
** Next steps: assert we can mock up tbpl to gather data post test completion, will mozilla be ok with that?
*** tbpl can do this: POST {'changeset': 31415926535, 'testname': 'dromaeo', 'platform': 'win7'} http://10.8.73.29/views/api/test_results
*** and recieve this: {'result':'PASS', 'tests': {'passed': 98, 'failed': 0, 'maybe': 2}, 'details': [{'k0s.org': {'actual': 2.71, 'expected': [2.75, 3.21]}, {'askjeads.com': {'actual': 99, 'expected': [88, 98]}]}

= Round Table =
* ctablert side note related to

= Action Items =
* [jmaher] - work with metrics to get tsvg going
* [jmaher] - build list of talos tests and find dev contacts 
** [jmaher] - ensure the dev contacts think the tests are useful and up to date or disable!
* [ctalbert] - set up developer meeting for July 10th at 2pm PDT
* [jeads] - get jmaher a list of os/branches/platforms that have been uploaded
** [jmaher] - work with that list to ensure we can match up with tbpl, recommend adjustments as necessary
* [jhammel] - own the documentation effort!
* [j*] - talos oauth 1.57 or something support for authentication to datazilla