Auto-tools/Goals/2014Q4

Top Level Goals

A list of high priority goals we're committed to achieving in Q4. These are also listed on the A-Team's Trello board.

Firefox

[DONE] Multi-binary devtools harness [ted]

Prototype a test harness which enables developers to run tests involving Firefox and another binary, such as Chrome, Safari, or a Firefox OS Device.

• why: So we can provide a mechanism to test devtools against targets other than self in continuous integration; currently most testing is semi-manual.
• bug 1064253
• Project page: TBD

[DONE] Mozmill e10s [jgriffin, dburns, ahal, chmanchester]

QA runs a valuable set of release tests on Firefox using Mozmill, which is not e10s compatible. To address this, we're going to enable the conversion of mozmill tests to Marionette, which will support e10s, and which has better ongoing support.

• why: so QA can continue running automated release and update tests of Firefox
• bug m21s
• Project page: https://wiki.mozilla.org/Auto-tools/Projects/m21s

Developer Productivity

[DONE] Autoland [dminor]

Deliver a functional prototype of the Autoland service that can be used for opt-in dogfooding. Full release to the entire community will occur after load testing and deployment in 2015Q1.

• Slight change to scope: we have a proof of concept that has been used to land to a clone of mozilla-central running on the Transplant service. Given the plans for Autoland in 2015 (interfacing with MozReview), there's no point in further work on this particular prototype.
• why: Developer productivity gains across the board, since this would reduce the amount of time required by developers to land a patch.
• bug 1043010
• Project page: https://wiki.mozilla.org/Auto-tools/Projects/Autoland

[DONE] Review Board [mcote]

Deploy Review Board for developers to start using (joint with RelEng) (carryover from Q3)

• why: across the board improvement in developer ergonomics for all gecko developers
• bug 1021929
• Project page: https://wiki.mozilla.org/Auto-tools/Projects/CodeReviewTool

[DEFER] Investigation of a wiki plugin for bug charts [ekyle]

Investigate the work needed to deliver a self-serve wiki feature to do simple bugzilla charting, like burn-down rates based on searches.

• deferred due to addition of goal to move dzAlerts to Treeherder
• why: Sustainability: make some types of charts self-serve, so we don't have to implement every request ourselves.
• bug: N/A
• Project page: will be created as a deliverable for this work.

Performance

[MISSED] Move dzAlerts from Datazilla to Treeherder [ekyle]

Make dzAlerts (which produces Talos performance regression alerts) ingest data from Treeherder, rather than Datazilla, as the latter doesn't scale well with Talos data.

• why: Improve the resilience of both Datazilla and dzAlerts.
• bug 1092206
• Project page: https://wiki.mozilla.org/Auto-tools/Projects/Alerts
• missed because Treeherder wasn't ingesting all of the Talos performance data, a fact which wasn't obvious until it was too late to address it in time to complete this goal

[DEFER] Create tests for the "charts" (dashboard) service [ekyle]

Create a set of unit tests for "charts", so that changes don't break existing functionality.

• deferred due to addition of goal to move dzAlerts to Treeherder
• why: Sustainability: charting code is getting complex due to all the branches created to avoid breaking other charts. Creating some tests would allow us to consolidate the branches by being confident that changes to the core JS code won't break tests. It will increase maintainability and time to develop new charts.
• bug 1080092
• Project page: https://wiki.mozilla.org/Auto-tools/Projects/Charts

[DONE] Make Talos compatible with e10s [wlach/jmaher]

Adapt Talos to work with Firefox in e10s mode, and get Talos tests running in e10s mode scheduled on trunk branches.

• why: Allow engineering to run Talos tests in e10s mode, so they can quantify performance issues.
• bug 1050706
• Project pages: https://wiki.mozilla.org/Electrolysis, https://wiki.mozilla.org/Buildbot/Talos

[DONE] Make Talos compatible with win64 [armenzg/jmaher]

Adapt Talos to work with win64 Firefox, and get Talos tests running against win64 scheduled on trunk branches.

• why: Allow engineering to run Talos tests in win64 mode, so they can quantify performance issues.
• bug 1080624
• Project pages: https://wiki.mozilla.org/Electrolysis, https://wiki.mozilla.org/Buildbot/Talos

[DONE] Monitor and file bugs for Talos performance regressions [jmaher]

We will monitor the Alert Manager for regressions in Talos tests and file bugs as needed. Sometime post-Q4 we will roll this into the normal sheriffing workflow, after we have the tooling to support that.

• why: To make performance regressions actionable.
• bug: N/A
• Project page: https://wiki.mozilla.org/Auto-tools/Projects/AlertManager

Treeherder

[DONE] Sheriff and developer improvements [jeads, mdoglio, camd, edmorley]

Continue to improve sheriff and developer workflows in Treeherder so we can effectively obsolete TBPL in the future.

• why: To make sheriffing easier with Treeherder and to make Treeherder a suitable replacement for TBPL for the general Mozilla community.
• bugs: bug 1059400
• Project page: https://wiki.mozilla.org/Auto-tools/Projects/Treeherder

Bugzilla

[DONE] Support for gmail transition [glob]

Identify and implement bugzilla features that are needed for the company to transition to using gmail as our mail service.

• why: To support the company's Gmail transition.
• bug: TBD
• Project page: https://mana.mozilla.org/wiki/display/IPPP/GMAIL+Project+Page

[MISSED] Prototype alternate bug views [glob]

Prototype some alternate bug views which can both decrease page load times and improve the user experience for specific use cases.

• why: Improve Bugzilla's speed and usability.
• bug: bug 1068655
• Project page: N/A

General Automation

[DONE] mochitest-chrome for B2G [gbrown]

Developers have requested that we add support for running the mochitest-chrome harness on FirefoxOS, so that they can write chrome-related tests without using hacky and problematic SpecialPowers workarounds in mochitest-plain. This goal does not include porting existing mochitest-chrome tests to B2G, just allowing new tests to be written using this harness.

• why: This would allow developers to write better mochitests for B2G.
• bug 797164
• Project page: N/A

[MISSED] Scope the consolidation of non-buildbot tests [bc]

We have several separate test suites that run on independent systems, like Autophone, Eideticker, mozbench, power profiling tests, etc. Each of these requires separate maintenance and feature development. We will scope the effort involved in consolidating different aspects of these, including test scheduling, test reporting, Android and Firefox OS device maintenance, and the use of buildbot for desktop tests, so we can determine whether or not we should attempt such a consolidation in 2015.

• why: Sustainability: each of our non-buildbot systems has a separate set of maintenance costs. By consolidating, we could reduce these, and potentially make the creation of new non-buildbot harnesses easier and faster.
• bug: N/A
• Project page: will be created as a deliverable for this work.
• missed because of work done (by request) to get video tests running in Autophone

[DONE] Make it possible to test mozharness changes on try [armenzg]

We want to be able to test mozharness changes on try, without having to run them on ash or cedar.

• why: faster development of mozharness-related code
• bug: bug 791924
• Project page: https://wiki.mozilla.org/User:Armenzg/Proposals/Mozharness_changes

[DEFER] Move more mozharness config info into the tree [armenzg]

note: deferred in favor of above goal of supporting mozharness-try

We want to move more config details that mozharness uses from the mozharness repo into the tree, e.g. run_file_names, specific_tests_zip_dirs, in_tree_config, all_$category_suites. The exact details moved will vary by script, but in general we'd like to move all the suite-specific details into the tree and out of mozharness.

• why: It helps us move faster by allowing us to test a greater number of things on try; it reduces the review burden on RelEng; it sets the stage for unifying mach and mozharness.
• bugs: bug 1067535, bug 1070041
• Project page: https://wiki.mozilla.org/User:Armenzg/Proposals/Mozharness_changes

Community

[DONE] A-Team Boot Camp [wlach]

Create a set of documentation to help onboard new contributors and others to working with the automation that the A-Team maintains.

• why: Make the A-Team more accessible to community members, and reduce the time we spend mentoring contributors on basic tasks.
• bug: N/A
• initial version available: http://ateam-bootcamp.readthedocs.org/en/latest/index.html

[DONE] Make our projects contributor-friendly [jmaher, jgriffin, all]

By the end of the quarter, all projects which are continuing to Q1 should be "contributor-friendly"...with a Project Page, up-to-date docs, etc.

• why: Make the A-Team's projects more accessible to community members, with the hope of increasing community engagement in more complex projects, and extending the span of a contributor's involvement.
• bug: N/A
• Project page: [Everything]

Supporting and Ongoing Tasks

A list of ongoing and supporting tasks that the A*Team is actively engaged in. The delivery date for these will vary depending on the resource requirements of our Top Level goals, as well as the amount of support requests that come in during the quarter, and their respective priorities.

See the list in our Trello board.

Backlog

We maintain a sizable backlog of tasks that we think are important to do, or that we've been asked to do, but for which we don't have resources to work on this quarter.

See the list in our Trello board.