Auto-tools/Projects/Autoland
Autoland
Introduction
Autoland is the name of a project which will enable automatic landing of changesets after a successful try run. This will allow developers to land changesets "automatically", with many fewer manual steps than are now required.
History
Several past projects have used the name "Autoland" and have had roughly the same goal. These have mostly failed to achieve deployment due to difficulty in being able to meaningfully determine whether a patch or changeset has "passed" a try run, due to the number of intermittent oranges which are likely to affect a given try push.
These projects include:
- BMO/AutoLand
- ReleaseEngineering:Autoland and related github repo
Non-goals
- This version of Autoland works on changesets, and will not retrieve patches out of Bugzilla.
- This version of Autoland will be designed to work against trunk branches, and will not be designed to assist in uplifting changesets. Future versions of Autoland may add that capability.
Bugs
Bugs should be filed in Bugzilla under Tree Management:Autoland.
Workflow
The proposed workflow for Autoland is as follows:
1 - ReviewBoard or developer submits patch/changeset to try with a specific syntax, maybe --autoland.
1a - If the try run is being requested by ReviewBoard, check that the requester (not the patch author) has commit access level 3, and if not, reject with an appropriate error message. If the try run has been requested directly, this check isn't performed until step 3.
2 - A try run is performed (with a subset of tests if specified by the submitter).
2a - When performed via ReviewBoard, we should encourage targeted try runs in order to conserve resources.
2b - When performed via ReviewBoard, we should force the try run against the tip of mozilla-inbound, in order to prevent the case where a changeset may pass a try run against an older gecko commit that the review was created against, but fail when landing on inbound.
3 - An Autoland service somewhere is consuming the pulse stream for the try repo, notices an Autoland job has been submitted to try and starts monitoring it.
3a - If the try run was submitted directly (not via ReviewBoard), the Autoland service should check the commit level access of the submitter, and reject the Autoland attempt if the submitter does not have commit access level 3.
4 - When no pending jobs remain, the Autoland service looks for failed jobs; if it finds any it retriggers them.
4a - Autoland will determine when all jobs are complete using the buildapi at [1]
4b - Autoland will determine job status by querying Treeherder
4c - Autoland will retrigger failed jobs using the buildapi
4d - Initially, Autoland will retrigger failed jobs only once before taking action; this can be tweaked in the future as needed if we want to make it smarter in hunting for frequent intermittent failures introduced by the changeset.
5 - When Autoland service gets results for the retriggered jobs:
5a - If all retriggered jobs pass, it lands the patch in-tree (m-i by default, by developer can request another tree using some TBD try syntax)
5b - If patch no longer cleanly applies, Autoland rejects with a message on the bug
5c - If the retriggered jobs fail, Autoland will annotate the push in Treeherder
6 - Sheriffs periodically monitor the Autoland queue in Treeherder for jobs that need human inspection, and flag them as ok to land or not ok to land, possibly after additional manual retriggers.
6a - If ok to land, autoland lands the patch in-tree. We may coalesce pending autoland commits and land them in batches periodically, like once per hour, ala Bumper Bot.
6b - If not ok to land, Autoland notifies the developer of the problem via bugzilla.
Components
Autoland Service [owner: dminor]
This is the primary component of Autoland. This service monitors pulse for autoland jobs submitted to try, monitors such try jobs for completion using the buildapi, queries Treeherder for job status, and retriggers failing jobs. It also submits comments to Bugzilla when the Autoland status of a try push changes, and lands patches on inbound trees if all success criteria are met.
Treeherder [owner: dminor or TBD]
Treeherder will be utilized in the following ways:
- The Autoland service will request job status (pass/fail) from Treeherder for Autoland try jobs
- Treeherder will expose a special Autoland view of try jobs which need sheriff attention; this view will include a button (available only to privileged users) that will allow a pending Autoland push to be landed or rejected.
Releng Buildapi
This api (which exists already) will be used to determine when Autoland try jobs are finished, and will be used to retrigger failing jobs.