Confirmed users
513
edits
Auto-tools/Projects/Alerts: Difference between revisions
→Datazilla Alerting System
Klahnakoski (talk | contribs) |
Klahnakoski (talk | contribs) |
||
| Line 7: | Line 7: | ||
dzAlerts' objective is to provide high quality, and detailed alerts on performance regressions. | dzAlerts' objective is to provide high quality, and detailed alerts on performance regressions. | ||
== Nomenclature == | == Nomenclature == | ||
| Line 33: | Line 19: | ||
* '''Score''' - The t-test and median test functions output a [http://en.wikipedia.org/wiki/P-value p-value]. This value is meaningful when it is close to zero, and should be charted on a vertical log-scale. For the sake of charting here we define SCORE = -log10(p-VALUE), which charts nicely alongside the time series, and provides a hint to the number of zeros to the right of the decimal place. | * '''Score''' - The t-test and median test functions output a [http://en.wikipedia.org/wiki/P-value p-value]. This value is meaningful when it is close to zero, and should be charted on a vertical log-scale. For the sake of charting here we define SCORE = -log10(p-VALUE), which charts nicely alongside the time series, and provides a hint to the number of zeros to the right of the decimal place. | ||
== | == Architecture == | ||
[[image:Auto-tools_Projects_Alerts_Architecture.png|right|thumb|200px]] | [[image:Auto-tools_Projects_Alerts_Architecture.png|right|thumb|200px]] | ||
* Inputs: | |||
** Datazilla web service provides test results | |||
** Datazilla copy of pushlog provides push dates | |||
* Outputs: | |||
** Alert database is used by [https://github.com/jmaher/alert_manager/blob/master/README.md Alert Manager] (incomplete) | |||
** Email alerts (consumed by sheriffs watching branches) | |||
The are two main parts to dzAlerts. The ETL step copies data from Datazilla to the ElasticSearch cluster, and the dzAlerts daemon is responsible for identification of performance regressions, and filling a database with what it finds. | The are two main parts to dzAlerts. The ETL step copies data from Datazilla to the ElasticSearch cluster, and the dzAlerts daemon is responsible for identification of performance regressions, and filling a database with what it finds. | ||
| Line 94: | Line 87: | ||
== Code == | == Code == | ||
https://github.com/klahnakoski/datazilla-alerts | https://github.com/klahnakoski/datazilla-alerts | ||
== Post Motrem (July 7th 2014) == | |||
dzAlerts started as code inside [https://wiki.mozilla.org/FirefoxOS/Performance/Datazilla Datazilla], hence the name. A few design changes were made during development: | |||
* The alerting code was distinct from the Datazilla code in that it was designed as stand alone deamons, with little integration with the main UI-centric code. | |||
* Integrating the alerts data access into the existing Datazilla database access pattern required too much boilerplate code that required additional maintenance as the project evolved. | |||
* The Datazilla database schema was not designed for pulling long time series on individual tests in a performant way. | |||
* The release schedules were different. | |||
dzAlerts is now completely separate, and uses an ElasticSearch cluster to pull slices out of the data cube of test results. It maintains enough information about the alerts so it can provide links to the Datazilla UI. | |||