Confirmed users
513
edits
Auto-tools/Projects/Alerts: Difference between revisions
→General Design
(→Code) |
Klahnakoski (talk | contribs) |
||
| Line 55: | Line 55: | ||
This same pattern happens in all tests of both dromaeo_css and dromaeo_dom. | This same pattern happens in all tests of both dromaeo_css and dromaeo_dom. | ||
=== Pulling Data === | |||
Requesting a time time series is done by simply filtering the test results by product, branch, platform and test name. ElasticSearch can pull months of Talos test results for any one combination in under a second. Each test is made of many replicates, and those replicates are reduced to a single statistic (usually median) and forwarded for analysis. | |||
=== Analysis === | |||
Once the test results are retrieved, and sorted by push date, several window functions are run over the data: They calculate the past stats, future stats, t-test pvalue, and median test pvalue, among other things. | |||
Decision of *if* there is a regression is done with the median test. The median test is insensitive to amplitude, which allows it to ignore the intermittent non-Gaussian noise we witness in many if the | |||
* what window size | |||
** currently: 20 data point before/after (globally adjustable, working on making it a per test parameter) | |||
* how to handle bimodal data (i.e. v8, dromaeo, etc.) | |||
** currently: using median as our metric mitigates some of the bimodal problem by naturally selecting the most common mode. | |||
** ideally: using a mixture model (http://en.wikipedia.org/wiki/Mixture_model) will help characterize the two modes and help us construct a useful aggregate | |||
dzAlerts uses the median test, and more points allow it to produce less false positives on (non-guassian) noisy data. The t-test is too sensitive to the non-gaussian noise we are witnessing and generating too many false alerts. | |||
== Math == | == Math == | ||