Confirmed users
2,177
edits
TestEngineering/Performance/Sheriffing/Alerts: Difference between revisions
TestEngineering/Performance/Sheriffing/Alerts (view source)
Revision as of 13:59, 29 January 2020
, 29 January 2020no edit summary
(more Autophone details) |
No edit summary |
||
| (9 intermediate revisions by 2 users not shown) | |||
| Line 2: | Line 2: | ||
== General triage process == | == General triage process == | ||
Perfherder is a performance monitoring tool, tracking many browser KPIs for each changeset. Although there are many KPIs which monitor very different things and in very different ways, Perfherder's UI is uniform and intuitive. One needn't know the specifics of a particular alert to start investigating it. Through means of retriggering and backfilling, it's pretty straightforward to narrow down on the offending changeset. Color hints are also easy to figure out: green always resembles performance improvements, while red means we now have a regression. | |||
== Types of alerts == | == Types of alerts == | ||
=== Talos === | === Talos === | ||
* short description: | * short description: Monitors various time-based performance KPIs on the browser ([https://wiki.mozilla.org/index.php?title=Performance_sheriffing/Talos/Tests more details]) | ||
* frequency: daily, little more than a dozen alerts | * frequency: daily, little more than a dozen alerts | ||
* [https://wiki.mozilla.org/ | * [https://wiki.mozilla.org/Performance_sheriffing/Talos/Sheriffing/Tree_FAQ#What_is_coalescing coalesced] by SETA?: yes (often requires backfilling) | ||
* available on platforms: | * available on platforms: | ||
** Windows: 7 32bit, 10 64bit (OPT, PGO builds) | ** Windows: 7 32bit, 10 64bit (OPT, PGO builds) | ||
| Line 15: | Line 15: | ||
** OS X: 10.10 (OPT builds only) | ** OS X: 10.10 (OPT builds only) | ||
* triaging specifics: | * triaging specifics: | ||
=== build_metrics === | === build_metrics === | ||
* short description: Monitor build times on multiple platforms, the size of the installers and other compiler-specific insights. | * short description: Monitor build times on multiple platforms, the size of the installers and other compiler-specific insights. | ||
* frequency: every 1-2 days, around 5 alerts | * frequency: every 1-2 days, around 5 alerts | ||
* contact: :froydnj, :ted.mielczarek, :gps | * contact: :froydnj, :ted.mielczarek, :gps | ||
* | * coalesced by SETA?: no (shouldn't require backfilling) | ||
* available on platforms: | * available on platforms: | ||
** Windows: 32/64bit (OPT, No-OPT, Mingw builds) | ** Windows: 32/64bit (OPT, No-OPT, Mingw builds) | ||
| Line 30: | Line 31: | ||
** when investigating, one should look for build config changes <ask :gps to provide more data> | ** when investigating, one should look for build config changes <ask :gps to provide more data> | ||
** build times often spike upwards for just a short time; they then lower to previous levels thanks to caching mechanisms set in place. We mark these as invalid alerts. | ** build times often spike upwards for just a short time; they then lower to previous levels thanks to caching mechanisms set in place. We mark these as invalid alerts. | ||
=== AWSY === | === AWSY === | ||
* short description: | * short description: Are We Slim Yet monitors memory consumption by the browser ([https://wiki.mozilla.org/AWSY/Tests more details]) | ||
* frequency: every 2-3 days, around half a dozen | * frequency: every 2-3 days, around half a dozen | ||
* contact: :erahm | * contact: :erahm | ||
* | * coalesced by SETA?: yes | ||
* available on platforms: | * available on platforms: | ||
** Windows: 32/64bit (OPT, PGO builds) | ** Windows: 32/64bit (OPT, PGO builds) | ||
| Line 55: | Line 44: | ||
* triaging specifics: | * triaging specifics: | ||
** retriggering/backfilling takes some time (>1h per test), so one must not abuse in collecting missing graph data | ** retriggering/backfilling takes some time (>1h per test), so one must not abuse in collecting missing graph data | ||
=== platform_microbench === | === platform_microbench === | ||
* short description: | * short description: N/A | ||
* frequency: daily, around 1-2 dozen alerts | * frequency: daily, around 1-2 dozen alerts | ||
* contact: | * contact: | ||
* | * coalesced by SETA?: yes | ||
* available on platforms: | * available on platforms: | ||
** Linux: 32bit (OPT builds), 64bit (OPT, PGO, ASAN builds) | ** Linux: 32bit (OPT builds), 64bit (OPT, PGO, ASAN builds) | ||