Confirmed users
656
edits
Sheriffing/Manifest Scheduling: Difference between revisions
no edit summary
No edit summary |
|||
| (8 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
The "manifest scheduling" project is a major shift in Mozilla's CI system. It aims to reduce costs and improve regression detection by only running the exact tests that we need. | The "manifest scheduling" project is a major shift in Mozilla's CI system. It aims to reduce costs and improve regression detection by only running the exact tests that we need. | ||
== Disable Manifest Scheduling == | |||
If manifest scheduling is enabled and it is causing problems that make it too difficult to do your job, please reach out to :ahal or :marco and we can disable it for you. If neither of us are around, you can disable it yourselves by removing this line: | |||
https://hg.mozilla.org/integration/autoland/file/9be5f086895c29015c576caab51921b0fa83c656/taskcluster/taskgraph/decision.py#l78 | |||
== Changes == | == Changes == | ||
| Line 27: | Line 33: | ||
The main issue is that under "manifest scheduling" the same mochitest-1 task on push A, will run a completely different set of tests as it does on push B. In other words, it will no longer be possible to filter Treeherder by task label to identify test-level regressions (though it should still work for many types of infra related issues). Instead, sheriffs will need to filter Treeherder by "test path". Read this [https://medium.com/@armenzg/filter-treeherder-jobs-by-test-or-manifest-path-af0e1ae74e61 blog post] for details of the feature. | The main issue is that under "manifest scheduling" the same mochitest-1 task on push A, will run a completely different set of tests as it does on push B. In other words, it will no longer be possible to filter Treeherder by task label to identify test-level regressions (though it should still work for many types of infra related issues). Instead, sheriffs will need to filter Treeherder by "test path". Read this [https://medium.com/@armenzg/filter-treeherder-jobs-by-test-or-manifest-path-af0e1ae74e61 blog post] for details of the feature. | ||
[[File:Test path.png | [[File:Test path.png|UI showing active filter for a test path]] | ||
[[File:Filtered tasks.png | [[File:Filtered tasks.png|Push showing tasks that executed the same test path]] | ||
Armen | Armen also added a new "Test Groups" to the job panel. It contains a list of manifests that the test task ran and each manifest will link to the test path filter outlined above. For example: | ||
[[File:Treeherder-test-groups-prototype.png]] | |||
=== Backfills === | === Backfills === | ||
| Line 41: | Line 45: | ||
Another major issue is backfilling. If tasks run different sets of tests on different pushes, then that will break the backfill action (as we'll need to make sure the exact same set of manifests were scheduled on each backfill push). | Another major issue is backfilling. If tasks run different sets of tests on different pushes, then that will break the backfill action (as we'll need to make sure the exact same set of manifests were scheduled on each backfill push). | ||
Thanks to work by Armen, the backfill action can now automatically detect if "manifest scheduling" was used for the task. If not it will perform the normal standard backfill you are used to. If so, it will run the same set of manifests from the originating push on all of the backfilled pushes. Because it's possible to run more than one backfill at a time, we need a way to identify which tasks were backfilled from where. To that end, the symbols of backfilled tasks have been changed to something like `<group>-bk(<symbol>-<rev>-bk)`. For example, if `M-fis(bc3)` was backfilled from revision `abcdef`, then the symbol for the backfill tasks would be `M-fis-bk(bc3-abcdef-bk)`. This notifies sheriffs that the task was backfilled starting at revision `abcdef` and contains the same set of test manifests as on that push. | |||
[[File:Backfilled tasks.png | [[File:Backfilled tasks.png|UI showing a failed task, two backfill requests and few retriggered tasks]] | ||
You can filter out a task you're backfilling and all backfilled tasks by selecting the task and selecting "Filter jobs containing these keywords" (text that shows once you hover the link). See the screenshot below for the location in the UI. | You can filter out a task you're backfilling and all backfilled tasks by selecting the task and selecting "Filter jobs containing these keywords" (text that shows once you hover the link). See the screenshot below for the location in the UI. | ||
[[File:Filter task and backfilled.png | [[File:Filter task and backfilled.png|Link to filter tasks and backfilled tasks]] | ||
Other than being aware of this change, there shouldn't be any differences in performing a backfill. | Other than being aware of this change, there shouldn't be any differences in performing a backfill. | ||
| Line 54: | Line 59: | ||
=== Add New Jobs === | === Add New Jobs === | ||
There is currently no way to specify test manifests when adding new jobs via Treeherder's "Add New Jobs" UI. This means that it can't be used to fill in tasks for the purpose of bisecting a regression. There are plans to add this feature in the future. | There is currently no way to specify test manifests when adding new jobs via Treeherder's "Add New Jobs" UI. This means that it can't be used to fill in tasks for the purpose of bisecting a regression. There are plans to add this feature in the future. If used, "Add New Jobs" would simply run the same manifests that ran | ||
One risk here is that it will be more difficult (but still possible) to schedule a mozilla-central only task on autoland for the purposes of bisecting a regression caught on central. You'll now need to make sure that you use "Add New Jobs" ''on a backstop push'' as that will run all test manifests. Then you can bisect the task like normal. | |||
In the future we plan to get rid of all mozilla-central only tasks and run them on an interval on autoland instead. | |||
=== Retriggers === | === Retriggers === | ||