Confirmed users
3,376
edits
Buildbot/Talos/Sheriffing: Difference between revisions
Jump to navigation
Jump to search
m
→What is an alert: - june 2016 update
m (→Overview: - update June 2016) |
m (→What is an alert: - june 2016 update) |
||
| Line 7: | Line 7: | ||
= What is an alert = | = What is an alert = | ||
As of January | As of January 2016, alerts are generated in [[https://treeherder.mozilla.org/perf.html#/alerts?status=0&framework=1 PerfHerder]]. These are generated by programatically verifying there is a sustained regression over time (original data point + [https://wiki.mozilla.org/Buildbot/Talos/Sheriffing/Noise_FAQ#Why_do_we_need_12_future_data_points 12 future data points]]). | ||
the | There is an alert summary outlining the alerts which match the same set of revisions. For the summary there are a few pieces of information: | ||
* | * Title (which is a good bug title if filing one for a regression: | ||
* platform | ** [https://wiki.mozilla.org/Buildbot/Talos/Sheriffing/Tree_FAQ#Branch_names_and_confusion branch] | ||
* | ** % regressed, this is a range of the regressions (not improvements) | ||
** the [https://wiki.mozilla.org/Buildbot/Talos/Tests tests] which have regressed | |||
** the platforms we see this regression on | |||
* date of the suspect revision push | |||
* link to the treeherder jobs view for the suspect revision | |||
* suspected changeset [range] including commit summary | |||
* a status field (default 'untriaged', where we can associate a bug, file a bug, and document the resolution. | |||
Below the summary will be a list of alerts, each alert will reference: | |||
* [https://wiki.mozilla.org/Buildbot/Talos/Tests Test name] | |||
* platform (including build type, such as opt, e10s, pgo) | |||
* old score (median score of the previous 12 commits) | |||
* new score (median score of the future 12 commits) | |||
* [[https://wiki.mozilla.org/Buildbot/Talos/Sheriffing/Alert_FAQ#Why_does_Alert_Manager_print_-xx.25 % change / values]] | * [[https://wiki.mozilla.org/Buildbot/Talos/Sheriffing/Alert_FAQ#Why_does_Alert_Manager_print_-xx.25 % change / values]] | ||
* | * bar chart to show severity, green = improvement, red = regression | ||
* | * Confidence value (from the t-test code) | ||
Keep in mind that alerts mention improvements and regressions, which is valuable for us to track the entire system as a whole. For filing bugs, we focus mostly on the regressions. | Keep in mind that alerts mention improvements and regressions, which is valuable for us to track the entire system as a whole. For filing bugs, we focus mostly on the regressions. | ||