Auto-tools/Projects/Alerts: Difference between revisions
Klahnakoski (talk | contribs) |
Klahnakoski (talk | contribs) |
||
| Line 30: | Line 30: | ||
* '''Test Suite''' - or "suite" for short, is a set of bundled tests. All the "tests" referred to by Datazilla and Talos are actually bundles of individual tests. ''Dromeao_css'', ''kraken'', ''tp5o'', and ''tscrollx'' are examples of test suites | * '''Test Suite''' - or "suite" for short, is a set of bundled tests. All the "tests" referred to by Datazilla and Talos are actually bundles of individual tests. ''Dromeao_css'', ''kraken'', ''tp5o'', and ''tscrollx'' are examples of test suites | ||
* '''Test''' - a single test responsible for testing a feature. Dromaeo_css has 6 tests called dojo.html, ext.html, jquery.html, mootools.html, prototype.html and yui.html. These can also be called "pages", or "page tests". | * '''Test''' - a single test responsible for testing a feature. Dromaeo_css has 6 tests called dojo.html, ext.html, jquery.html, mootools.html, prototype.html and yui.html. These can also be called "pages", or "page tests", depending on the suite. | ||
* '''Sub-Test''' - A test should be the finest grain, but unfortunately some tests bundle yet-deeper-but-unnamed tests into a single result. These are ETLed into normal tests for dzAlerts to digest, but we need a name for these in the wild. | * '''Sub-Test''' - A test should be the finest grain, but unfortunately some tests bundle yet-deeper-but-unnamed tests into a single result. These are ETLed into normal tests for dzAlerts to digest, but we need a name for these in the wild. | ||
* '''Replicate''' - Each test is executed (replicated) multiple times, and the result stored in an array. Each result is called a "replicate", and the array of results is usually referred to as "the replicates". | * '''Replicate''' - Each test is executed (replicated) multiple times, and the result stored in an array. Each result is called a "replicate", and the array of results is usually referred to as "the replicates". | ||
Revision as of 15:00, 12 June 2014
Datazilla Alerting System
The Datazilla Alerting System, also known as dzAlerts. Is a daemon that periodically inspects the Talos, B2G, and Eideticker performance results and generates alerts when there is a statistically significant regression in performance.
Collaboration
Features and development are ongoing
- Signal from Noise meeting (notes on etherpad)
- IRC: #ateam@irc.mozilla.org
- email: klahnakoski@mozilla.com
Objective
dzAlerts' objective is to provide high quality, and detailed alerts on performance regressions.
History
dzAlerts started as code inside Datazilla, hence the name. A few design changes were made during development:
- The alerting code was distinct from the Datazilla code in that it was designed as stand alone deamons, with little integration with the main UI-centric code.
- Integrating the alerts data access into the existing Datazilla database access pattern required too much boilerplate code that required additional maintenance as the project evolved.
- The Datazilla database schema was not designed for pulling long time series on individual tests in a performant way.
- The release schedules were different.
dzAlerts is now completely separate, and uses an ElasticSearch cluster to pull slices out of the data cube of test results. It maintains enough information about the alerts so it can provide links to the Datazilla UI.
Nomenclature
The nomenclature used by dzAlerts is a little different than the surrounding applications it deals with. This is to be consistent with the definitions of the same name overall, and to provide space for naming the fine features dzAlerts has visibility to, but are invisible to other systems.
- Test Suite - or "suite" for short, is a set of bundled tests. All the "tests" referred to by Datazilla and Talos are actually bundles of individual tests. Dromeao_css, kraken, tp5o, and tscrollx are examples of test suites
- Test - a single test responsible for testing a feature. Dromaeo_css has 6 tests called dojo.html, ext.html, jquery.html, mootools.html, prototype.html and yui.html. These can also be called "pages", or "page tests", depending on the suite.
- Sub-Test - A test should be the finest grain, but unfortunately some tests bundle yet-deeper-but-unnamed tests into a single result. These are ETLed into normal tests for dzAlerts to digest, but we need a name for these in the wild.
- Replicate - Each test is executed (replicated) multiple times, and the result stored in an array. Each result is called a "replicate", and the array of results is usually referred to as "the replicates".
General Design
The are two main parts to dzAlerts. The ETL step copies data from Datazilla to the ElasticSearch cluster, and the dzAlerts daemon is responsible for identification of performance regressions, and filling a database with what it finds.
ETL
The Extract Transform Load (ETL) daemon is responsible for
- Extracting the JSON test suite results from Datazilla,
- Transforming the bundle of test results into individual records, and
- Loading them into the ElasticSearch cluster.
The transform step is the most interesting, but still simple: In the case of tp5o, of the 50 page tests results is given it's own record with a copy of all the tp5o metadata (like platform, branch, test_run_time, etc). This is highly redundant, but allows ElasticSearch to index each test simply for fast retrieval.
Dromeao* ETL
The two dromaeo* suites (dromaeo_css, dromaeo_dom) are a little more complicated, and best demonstrated with couple charts from Datazilla. The top chart shows modify.html test mean and variance. One of the tests is highlighted, and you can see the individual replicates in the bottom chart. Looking at the replicates, we see they are in sets-of-five: Each five replicates has approximately the same value, and each set-of-five seems distinct from the others. Now that you see the pattern in the replicates, you can understand why the variance shown in the above chart is consistently high.
This same pattern happens in all tests of both dromaeo_css and dromaeo_dom.