Engineering Workflow/Road Map: Difference between revisions
(include revision-stack landings in "Lando for All") |
(Commit series work) |
||
| (4 intermediate revisions by the same user not shown) | |||
| Line 5: | Line 5: | ||
== Current == | == Current == | ||
=== Phabricator | === Better support for commit series in Phabricator and Lando === | ||
The | The official command-line interface for Phabricator, Arcanist, does not meet the needs of engineering when it comes to first-class support for submitting, updating, and pulling down series of commits. Lando also does not currently support the atomic landing of an entire series of commits. We will build a new command-line interface to Phabricator and extend Lando to better support standard Firefox engineering practices. | ||
Impact: | Impact: | ||
* | * Efficiently submit and update commit series, encouraging smaller commits. | ||
* Smoothly land commit series, avoiding repetitive clicking and the possibility of merge conflicts. | |||
* | * Eventually, ease the contribution pipeline by not requiring the installation of Arcanist, which can be annoying on some platforms (primarily Windows). | ||
* | |||
=== | === BMO Auth0 Integration === | ||
Our Bugzilla instance will be tied into Mozilla's Auth0 IAM system, which will allow the linking of BMO accounts to LDAP, Mozillians, etc. | |||
Impact: | Impact: | ||
* | * Easily identify employees, NDAed volunteers, and other contributor groups. | ||
* | * Applications integrated with BMO, e.g. Phabricator, can access users' LDAP accounts and other Mozillian data. | ||
* Applications already integrated into Mozilla's IAM system can access Bugzilla user data. | |||
* | |||
=== Vendor manifests === | === Vendor manifests === | ||
| Line 72: | Line 60: | ||
== Next == | == Next == | ||
=== | === Transplant internal improvements === | ||
The Transplant service is the engine that takes diffs from Phabricator (via Lando) and, until its retirement, MozReview and lands them in the destination repository on behalf of a user. There is some design debt in its architecture and deployability that will benefit future work on related services (particularly Lando). | |||
Impact: | Impact: | ||
* | * Lower maintenance costs. | ||
* | * Easier to develop locally. | ||
* | * Better support from operations. | ||
=== Lando for All === | === Lando for All === | ||
| Line 96: | Line 84: | ||
=== Intent-to-land === | === Intent-to-land === | ||
=== cinnabar-less Git | === Vanilla (cinnabar-less) Git support for mozilla-central === | ||
git-cinnabar was created to allow developers to use Git to interact with Mercurial, mainly [https://hg.mozilla.org|hg.mozilla.org]. It has some downsides, however, such as being a bit tricky to install. Ideally developers would be able to use vanilla Git to contribute to Firefox. | |||
If we view Phabricator and Lando as a conduit to getting code into mozilla-central, then ultimately VCS clients would only need to talk to Phabricator. Phabricator natively supports both Mercurial and Git<sup>[[#vcs-parent-note|1]]</sup>, and full Lando support means that developers will not need to talk to hg.mozilla.org directly. We just need to ensure that the user experience of submitting a patch in progress to Try remains as straightforward as it is today. | |||
Impact: | |||
* No need to maintain nor educate people on git-cinnabar. | |||
* End to unproductive VCS debates. | |||
* Solidifies Phabricator-Lando as the conduit to hg.mozilla.org; this consolidation will make it easier to add more services and automation to the commit pipeline. | |||
<div id="vcs-parent-note><small> | |||
1. There is one wrinkle in supporting two VCSes for the same repository: like figuring out parent commits when a different VCS is used to pull down commits; see {{bug|1443375}} for how this was solved with git-cinnabar.</small></div> | |||
=== Lando headless client API support === | === Lando headless client API support === | ||
=== Lando and Phabricator backout support === | === Lando and Phabricator backout support === | ||
=== Run tests from source directory in local and ci builds === | === Run tests from source directory in local and ci builds === | ||
| Line 113: | Line 111: | ||
=== Artifact caching experiments for builds === | === Artifact caching experiments for builds === | ||
== Done == | |||
=== Phabricator integration === | |||
The bulk of Phabricator integration into BMO was completed in May. Phabricator is now available for general usage. | |||
Impact: | |||
* One sole tool for review allows the Engineering Workflow to concentrate its efforts, and eliminates the need for contributors to learn several tools. | |||
* A more intuitive interface than BMO and MozReview, with a contemporary look to appeal to new contributors. | |||
* Some integration with existing tools and processes, balancing complexity and disruption. | |||
* Excellent support from Phabricator's developers ([https://phacility.com Phacility]) and a large user community. | |||
=== Lando initial release === | |||
[https://lando.services.mozilla.com Lando] is the new automatic code-landing tool integrated with Phabricator. It is functionally similar to MozReview's Autoland but is designed to be more maintainable and extensible by being less tightly coupled to the review tool. | |||
Impact: | |||
* One-button landing offers an easier way to land code and minimal context switching from reviewing code. | |||
* Queued landings due to tree closures means developers don't have to be around when they open to land their code. | |||
* Becomes a common gateway to mozilla-central (and other repos) which allows more process and security improvements (see below). | |||
Latest revision as of 01:47, 12 July 2018
Engineering Workflow Road Map
These are the improvements and new services we are currently working on and others that we are considering for the near future (1-2 years). It is subject to change.
Current
Better support for commit series in Phabricator and Lando
The official command-line interface for Phabricator, Arcanist, does not meet the needs of engineering when it comes to first-class support for submitting, updating, and pulling down series of commits. Lando also does not currently support the atomic landing of an entire series of commits. We will build a new command-line interface to Phabricator and extend Lando to better support standard Firefox engineering practices.
Impact:
- Efficiently submit and update commit series, encouraging smaller commits.
- Smoothly land commit series, avoiding repetitive clicking and the possibility of merge conflicts.
- Eventually, ease the contribution pipeline by not requiring the installation of Arcanist, which can be annoying on some platforms (primarily Windows).
BMO Auth0 Integration
Our Bugzilla instance will be tied into Mozilla's Auth0 IAM system, which will allow the linking of BMO accounts to LDAP, Mozillians, etc.
Impact:
- Easily identify employees, NDAed volunteers, and other contributor groups.
- Applications integrated with BMO, e.g. Phabricator, can access users' LDAP accounts and other Mozillian data.
- Applications already integrated into Mozilla's IAM system can access Bugzilla user data.
Vendor manifests
Provide a common structure, process, and tools for managing third-party code in mozilla-central. An Intent to Require was posted to dev.platform on 2018/04/10 and finalized on 2018/10/18. Follow the tracking bug (bug 1454867) for progress.
Impact:
- Easier to identify and find third-party code in mozilla-central.
- Easier to determine metadata about third-party code, including original source and current revision.
- Easier to update third-party code.
BMO usability
Implement a number of long-awaited UX enhancements, including MarkDown, full UTF-8 support, and quick edits.
Impact:
- User delight!
- Less friction for new users.
- Better expressibility and accuracy in comments.
Hg scalability and efficiency
Implement hgweb mirroring in AWS and upstream remotefilelog extension for hg
Impact:
- Multi-region redundancy and autoscaling for hgweb servers for additional resiliency.
- Faster checkouts to enable shorter end to end times for local and CI builds.
Build backend
Implement a new build backend for both local and CI builds. Deploy a new distributed sccache system written in Rust to enable fast, secure local builds in offices. Implement local build telemetry that submits data to the generic data ingestion pipeline to allow us to identify and reduce inefficiencies in local builds.
Impact:
- Reduce the incidence of clobber builds.
- Faster incremental builds.
- Secure and IT supported mechanism for faster local builds in offices.
Next
Transplant internal improvements
The Transplant service is the engine that takes diffs from Phabricator (via Lando) and, until its retirement, MozReview and lands them in the destination repository on behalf of a user. There is some design debt in its architecture and deployability that will benefit future work on related services (particularly Lando).
Impact:
- Lower maintenance costs.
- Easier to develop locally.
- Better support from operations.
Lando for All
The initial version of Lando will miss three important features required to be a complete replace for pushing directly to Firefox repositories: support for confidential revisions, and Try, and revision stacks. The latter two were included in MozReview's Autoland system. The former was notably lacking in MozReview.
Impact:
- Efficiency: users can easily send off a Try job from a Phabricator revision and can easily land multiple revisions at once.
- Clarity of tooling and better maintainability, as we can then eliminate patch review in BMO.
- Information clarity: discoverable record of results of Try pushes for a given revision.
Future
Enforced reviewers
Intent-to-land
Vanilla (cinnabar-less) Git support for mozilla-central
git-cinnabar was created to allow developers to use Git to interact with Mercurial, mainly [1]. It has some downsides, however, such as being a bit tricky to install. Ideally developers would be able to use vanilla Git to contribute to Firefox.
If we view Phabricator and Lando as a conduit to getting code into mozilla-central, then ultimately VCS clients would only need to talk to Phabricator. Phabricator natively supports both Mercurial and Git1, and full Lando support means that developers will not need to talk to hg.mozilla.org directly. We just need to ensure that the user experience of submitting a patch in progress to Try remains as straightforward as it is today.
Impact:
- No need to maintain nor educate people on git-cinnabar.
- End to unproductive VCS debates.
- Solidifies Phabricator-Lando as the conduit to hg.mozilla.org; this consolidation will make it easier to add more services and automation to the commit pipeline.
Lando headless client API support
Lando and Phabricator backout support
Run tests from source directory in local and ci builds
Overhauling release repositories and build configurations to use named branches
Support for platforms other than Linux for new build backend
Cross-compile a Windows build with clang-cl from Linux
Artifact caching experiments for builds
Done
Phabricator integration
The bulk of Phabricator integration into BMO was completed in May. Phabricator is now available for general usage.
Impact:
- One sole tool for review allows the Engineering Workflow to concentrate its efforts, and eliminates the need for contributors to learn several tools.
- A more intuitive interface than BMO and MozReview, with a contemporary look to appeal to new contributors.
- Some integration with existing tools and processes, balancing complexity and disruption.
- Excellent support from Phabricator's developers (Phacility) and a large user community.
Lando initial release
Lando is the new automatic code-landing tool integrated with Phabricator. It is functionally similar to MozReview's Autoland but is designed to be more maintainable and extensible by being less tightly coupled to the review tool.
Impact:
- One-button landing offers an easier way to land code and minimal context switching from reviewing code.
- Queued landings due to tree closures means developers don't have to be around when they open to land their code.
- Becomes a common gateway to mozilla-central (and other repos) which allows more process and security improvements (see below).