342
edits
TestEngineering/Performance/Sheriffing/Tree FAQ: Difference between revisions
TestEngineering/Performance/Sheriffing/Tree FAQ (view source)
Revision as of 13:54, 16 January 2020
, 16 January 2020Redirected page to Performance FAQ
(improve expression) |
Alexandrui (talk | contribs) (Redirected page to Performance FAQ) |
||
| (5 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
#REDIRECT [[Performance FAQ|https://wiki.mozilla.org/TestEngineering/Performance/FAQ#Sheriffing]] | |||
= Branch names and confusion = | = Branch names and confusion = | ||
We have a variety of branches at Mozilla, here are the main ones that we see alerts on: | We have a variety of branches at Mozilla, here are the main ones that we see alerts on: | ||
| Line 5: | Line 6: | ||
* Mozilla-Beta (all PGO) | * Mozilla-Beta (all PGO) | ||
Linux and Windows builds have [ | Linux and Windows builds have [[TestEngineering/Performance/Sheriffing/Tree_FAQ#What_is_PGO|PGO]], OSX does not. | ||
When investigating alerts, always look for the Non-PGO branch first. Usually expect to find changes on Mozilla-Inbound (about 50%) and Autoland (50%). | When investigating alerts, always look for the Non-PGO branch first. Usually expect to find changes on Mozilla-Inbound (about 50%) and Autoland (50%). | ||
| Line 22: | Line 23: | ||
* missing builds - we would have to generate builds, which automatically schedules test jobs, sometimes these test jobs are coalesced and not run. | * missing builds - we would have to generate builds, which automatically schedules test jobs, sometimes these test jobs are coalesced and not run. | ||
* results might not be possible due to build failures, or test failures | * results might not be possible due to build failures, or test failures | ||
* [[ | * [[TestEngineering/Performance/Sheriffing/Tree_FAQ#What_is_PGO|pgo builds]] are not coalesced, they just run much less frequently. Most likely a pgo build isn't the root cause | ||
Here is a view on treeherder of missing data (usually coalescing): | Here is a view on treeherder of missing data (usually coalescing): | ||
| Line 31: | Line 32: | ||
= What is an uplift = | = What is an uplift = | ||
Every [ | Every [[RapidRelease/Calendar|6 weeks]] we release a new version of Firefox. When we do that, our code which developers check into the nightly branch gets uplifted (thing of this as a large [[TestEngineering/Performance/Sheriffing/Tree_FAQ#What_is_a_merge|merge]]) to the Beta branch. Now all the code, features, and Talos regressions are on Beta. | ||
This affects the Performance Sheriffs because we will get a big pile of alerts for Mozilla-Beta. These need to be addressed rapidly. Luckily almost all the regressions seen on Mozilla-Beta will already have been tracked on Mozilla-Inbound or Autoland. | This affects the Performance Sheriffs because we will get a big pile of alerts for Mozilla-Beta. These need to be addressed rapidly. Luckily almost all the regressions seen on Mozilla-Beta will already have been tracked on Mozilla-Inbound or Autoland. | ||
= What is a merge = | = What is a merge = | ||
Many times each day we merge code from the integration branches into the main branch and back. This is a common process in large projects. At Mozilla, this means that the majority of the code for Firefox is checked into Mozilla-Inbound and Autoland, then it is merged into Mozilla-Central (also referred to as Firefox) and then once merged, it gets merged back into the other branches. If you want to read more about this merge procedure, here are [ | Many times each day we merge code from the integration branches into the main branch and back. This is a common process in large projects. At Mozilla, this means that the majority of the code for Firefox is checked into Mozilla-Inbound and Autoland, then it is merged into Mozilla-Central (also referred to as Firefox) and then once merged, it gets merged back into the other branches. If you want to read more about this merge procedure, here are [[Sheriffing/How_To/Merges|the details]]. | ||
Here is an example of a view of what a merge looks like on [https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=126a1ec5c7c5 TreeHerder]: | Here is an example of a view of what a merge looks like on [https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=126a1ec5c7c5 TreeHerder]: | ||
| Line 49: | Line 50: | ||
= What is a backout = | = What is a backout = | ||
Many times we backout or hotfix code as it is causing a build failure or unittest failure. The [ | Many times we backout or hotfix code as it is causing a build failure or unittest failure. The [[Sheriffing/Sheriff_Duty|Sheriff team]] handles this process in general and backouts/hotfixes are usually done within 3 hours (i.e. we won't have [[TestEngineering/Performance/Sheriffing/Noise_FAQ#Why_do_we_need_12_future_data_points|12 future changesets]]) of the original fix. As you can imagine we could get an alert 6 hours later and go to look at the graph and see there is no regression, instead there is a temporary spike for a few data points. | ||
While looking on TreeHerder for a backout, they all mention a backout in the commit message: | While looking on TreeHerder for a backout, they all mention a backout in the commit message: | ||
| Line 57: | Line 58: | ||
* note ^ the above image mentions the bug that was backed out, sometimes it is the revisoin | * note ^ the above image mentions the bug that was backed out, sometimes it is the revisoin | ||
Backouts which affect | Backouts which affect [[TestEngineering/Performance/Sheriffing/Alerts|Perfherder alerts]] always generate a set of improvements and regressions. These are usually easy to spot on the graph server and we just need to annotate the set of alerts for the given revision to be a 'backout' with the bug to track what took place. | ||
Here is a view on graph server of what appears to be a backout (it could be a fix that landed quickly also): | Here is a view on graph server of what appears to be a backout (it could be a fix that landed quickly also): | ||
| Line 73: | Line 74: | ||
* OSX does not do PGO builds, so we do not have -Non-PGO branches for those platforms. (i.e. we only have Mozilla-Inbound) | * OSX does not do PGO builds, so we do not have -Non-PGO branches for those platforms. (i.e. we only have Mozilla-Inbound) | ||
* PGO alerts will probably have different regression percentages, but the overall list of platforms/tests for a given revision will be almost identical | * PGO alerts will probably have different regression percentages, but the overall list of platforms/tests for a given revision will be almost identical | ||