160
edits
TestEngineering/Performance/Sheriffing/Tree FAQ: Difference between revisions
Jump to navigation
Jump to search
TestEngineering/Performance/Sheriffing/Tree FAQ (view source)
Revision as of 12:39, 22 January 2018
, 22 January 2018update links
(update paragraph) |
(update links) |
||
| Line 5: | Line 5: | ||
* Mozilla-Beta (all PGO) | * Mozilla-Beta (all PGO) | ||
Linux and Windows builds have [https://wiki.mozilla.org/ | Linux and Windows builds have [https://wiki.mozilla.org/Performance_sheriffing/Talos/Sheriffing/Tree_FAQ#What_is_PGO PGO], OSX does not. | ||
When investigating alerts, always look for the Non-PGO branch first. Usually expect to find changes on Mozilla-Inbound (about 50%) and Autoland (50%). | When investigating alerts, always look for the Non-PGO branch first. Usually expect to find changes on Mozilla-Inbound (about 50%) and Autoland (50%). | ||
| Line 22: | Line 22: | ||
* missing builds - we would have to generate builds, which automatically schedules test jobs, sometimes these test jobs are coalesced and not run. | * missing builds - we would have to generate builds, which automatically schedules test jobs, sometimes these test jobs are coalesced and not run. | ||
* results might not be possible due to build failures, or test failures | * results might not be possible due to build failures, or test failures | ||
* [[https://wiki.mozilla.org/ | * [[https://wiki.mozilla.org/Performance_sheriffing/Talos/Sheriffing/Tree_FAQ#What_is_PGO pgo builds]] are not coalesced, they just run much less frequently. Most likely a pgo build isn't the root cause | ||
Here is a view on treeherder of missing data (usually coalescing): | Here is a view on treeherder of missing data (usually coalescing): | ||
| Line 31: | Line 31: | ||
= What is an uplift = | = What is an uplift = | ||
Every [https://wiki.mozilla.org/RapidRelease/Calendar 6 weeks] we release a new version of Firefox. When we do that, our code which developers check into the nightly branch gets uplifted (thing of this as a large [https://wiki.mozilla.org/ | Every [https://wiki.mozilla.org/RapidRelease/Calendar 6 weeks] we release a new version of Firefox. When we do that, our code which developers check into the nightly branch gets uplifted (thing of this as a large [https://wiki.mozilla.org/Performance_sheriffing/Talos/Sheriffing/Tree_FAQ#What_is_a_merge merge]) to the Beta branch. Now all the code, features, and Talos regressions are on Beta. | ||
This affects the Performance Sheriffs because we will get a big pile of alerts for Mozilla-Beta. These need to be addressed rapidly. Luckily almost all the regressions seen on Mozilla-Beta will already have been tracked on Mozilla-Inbound or Autoland. | This affects the Performance Sheriffs because we will get a big pile of alerts for Mozilla-Beta. These need to be addressed rapidly. Luckily almost all the regressions seen on Mozilla-Beta will already have been tracked on Mozilla-Inbound or Autoland. | ||
| Line 49: | Line 49: | ||
= What is a backout = | = What is a backout = | ||
Many times we backout or hotfix code as it is causing a build failure or unittest failure. The [https://wiki.mozilla.org/Sheriffing/Sheriff_Duty Sheriff team] handles this process in general and backouts/hotfixes are usually done within 3 hours (i.e. we won't have [https://wiki.mozilla.org/ | Many times we backout or hotfix code as it is causing a build failure or unittest failure. The [https://wiki.mozilla.org/Sheriffing/Sheriff_Duty Sheriff team] handles this process in general and backouts/hotfixes are usually done within 3 hours (i.e. we won't have [https://wiki.mozilla.org/Performance_sheriffing/Talos/Sheriffing/Noise_FAQ#Why_do_we_need_12_future_data_points 12 future changesets]) of the original fix. As you can imagine we could get an alert 6 hours later and go to look at the graph and see there is no regression, instead there is a temporary spike for a few data points. | ||
While looking on TreeHerder for a backout, they all mention a backout in the commit message: | While looking on TreeHerder for a backout, they all mention a backout in the commit message: | ||
| Line 73: | Line 73: | ||
* OSX does not do PGO builds, so we do not have -Non-PGO branches for those platforms. (i.e. we only have Mozilla-Inbound) | * OSX does not do PGO builds, so we do not have -Non-PGO branches for those platforms. (i.e. we only have Mozilla-Inbound) | ||
* PGO alerts will probably have different regression percentages, but the overall list of platforms/tests for a given revision will be almost identical | * PGO alerts will probably have different regression percentages, but the overall list of platforms/tests for a given revision will be almost identical | ||
* [https://wiki.mozilla.org/index.php?title= | * [https://wiki.mozilla.org/index.php?title=Performance_sheriffing/Talos/Sheriffing/Tree_FAQ&action=edit duplicated & updated from old page] | ||