2
edits
TestEngineering/Performance/Sheriffing/Workflow: Difference between revisions
Jump to navigation
Jump to search
TestEngineering/Performance/Sheriffing/Workflow (view source)
Revision as of 20:55, 5 January 2024
, 5 January 2024→Harness alerts: adding section for baseline changes
(Added section for how to become a sheriff) |
(→Harness alerts: adding section for baseline changes) |
||
| Line 151: | Line 151: | ||
=== Harness alerts === | === Harness alerts === | ||
Harness alerts are usually caused by re-recordings or changes to code from testing/raptor component changing the baseline. If the regression is assumed, then link the culprit directly to the alert, close as WONTFIX and add the "harness" tag. Otherwise, open a regression bug. | Harness alerts are usually caused by re-recordings or changes to code from testing/raptor component changing the baseline. If the regression is assumed, then link the culprit directly to the alert, close as WONTFIX and add the "harness" tag. Otherwise, open a regression bug. | ||
===== Baseline Changes ===== | |||
Baseline changes are hard to identify, but we have some questions we can ask ourselves and with other people on the team to help determine if we are dealing with a real regression or a baseline change: | |||
1) Should the patch that caused a change in the baseline metrics because we are aiming for testing in a different environment? | |||
If yes, then the new baseline may be accepted. As the conditions for testing have changed we have to expect that some things will change. Take for instance adding a blank profile to the conditioned profile changes, as this is testing in a different environment we expect things will change | |||
2) Is the patch changing something that shouldn't cause a shift in metrics? | |||
If yes, then do not accept the new baseline immediately, do an analysis and discuss with other sheriffs to get second opinions. | |||
3) Does the patch change the test itself? | |||
If yes, a new baseline may be accepted depending on the changes. If the changes are not functional, then we shouldn’t accept a new baseline. | |||
4) Are test recordings changing? | |||
If yes, then a new baseline should be accepted after checking to make sure that the page that is being tested is different from the previous recording. If they aren’t, then the differences should be investigated further - it’s possible that they still may be a valid baseline change even if the page is the same, as non-visual changes may be different. | |||
5) Any other cases that do not fall in these categories should be discussed with the team for second opinions and with patch authors if needed. | |||
=== Backout/regression-fix alerts === | === Backout/regression-fix alerts === | ||