Confirmed users
574
edits
Telemetry/Alerts: Difference between revisions
m
Normalize heading levels
(linkification and name-checking Bhattacharyya) |
m (Normalize heading levels) |
||
| Line 1: | Line 1: | ||
=Telemetry Alerts= | = Telemetry Alerts = | ||
Many Telemetry probes were created to show performance trends over time. Sudden changes happening in Nightly could be the sign of an unintentional performance regression, so we introduced a system to automatically detect and alert developers about such changes. | Many Telemetry probes were created to show performance trends over time. Sudden changes happening in Nightly could be the sign of an unintentional performance regression, so we introduced a system to automatically detect and alert developers about such changes. | ||
Thus we created Telemetry Alerts. It comes in two pieces: [https://github.com/mozilla/cerberus/ Cerberus] the Detector and [https://github.com/mozilla/medusa/ Medusa] the Front-end. | Thus we created Telemetry Alerts. It comes in two pieces: [https://github.com/mozilla/cerberus/ Cerberus] the Detector and [https://github.com/mozilla/medusa/ Medusa] the Front-end. | ||
== Cerberus == | |||
Every day Cerberus grabs the latest aggregated information about all non-keyed Telemetry probes from aggregates.telemetry.mozilla.org and compares the distribution of values from the '''Nightly''' builds of the past two days to the distribution of values from the Nightly builds of the past seven days. | Every day Cerberus grabs the latest aggregated information about all non-keyed Telemetry probes from aggregates.telemetry.mozilla.org and compares the distribution of values from the '''Nightly''' builds of the past two days to the distribution of values from the Nightly builds of the past seven days. | ||
| Line 11: | Line 11: | ||
It places all detected changes in a file for ingestion by Medusa. | It places all detected changes in a file for ingestion by Medusa. | ||
== Medusa == | |||
Medusa is in charge of emailing people when distributions change and for displaying the website https://alerts.telemetry.mozilla.org which contains pertinent information about each detected regression. | Medusa is in charge of emailing people when distributions change and for displaying the website https://alerts.telemetry.mozilla.org which contains pertinent information about each detected regression. | ||
Medusa also checks for expiring histograms and sends emails notifying of their expiry. | Medusa also checks for expiring histograms and sends emails notifying of their expiry. | ||
==What it can do== | == What it can do == | ||
Telemetry Alerts is very good at identifying sudden changes in the shapes of normalized distributions of Telemetry probes. If you can see the distribution of [https://mzl.la/2vdMRax GC_MS] shift from one day to the next, then likely so can Cerberus. | Telemetry Alerts is very good at identifying sudden changes in the shapes of normalized distributions of Telemetry probes. If you can see the distribution of [https://mzl.la/2vdMRax GC_MS] shift from one day to the next, then likely so can Cerberus. | ||
==What can't it do== | == What can't it do == | ||
Telemetry Alerts is not able to see sudden shifts in volume. It is also very easily fooled if a change happens over a long period of time or doesn't fundamentally alter the shape of the probe's histogram. | Telemetry Alerts is not able to see sudden shifts in volume. It is also very easily fooled if a change happens over a long period of time or doesn't fundamentally alter the shape of the probe's histogram. | ||
| Line 27: | Line 27: | ||
* We suddenly received twice as many pings from 200-tab subsessions (the dominance of 1-tab pings would likely ensure the overall shape of the distribution changed insufficiently much for Cerberus to pick up on it) | * We suddenly received twice as many pings from 200-tab subsessions (the dominance of 1-tab pings would likely ensure the overall shape of the distribution changed insufficiently much for Cerberus to pick up on it) | ||
==Telemetry Alert Emails== | == Telemetry Alert Emails == | ||
One of the main ways humans interact with Telemetry Alerts is through the emails sent by Medusa. | One of the main ways humans interact with Telemetry Alerts is through the emails sent by Medusa. | ||
| Line 34: | Line 33: | ||
== Triaging a Telemetry Alert Email == | == Triaging a Telemetry Alert Email == | ||
Congratulations! You have just received a Telemetry Alert! | Congratulations! You have just received a Telemetry Alert! | ||