Confirmed users
1,927
edits
Auto-tools/Goals/2013Q3: Difference between revisions
m
→P0: Bugzilla
m (→P0: Bugzilla) |
|||
| (33 intermediate revisions by 9 users not shown) | |||
| Line 13: | Line 13: | ||
** {{done|Built in-cloud on AWS}} | ** {{done|Built in-cloud on AWS}} | ||
** {{defer|Integration with Treeherder}} | ** {{defer|Integration with Treeherder}} | ||
** {{ | ** {{done|Allows tests to be defined simply in the main m-c tree}} | ||
** {{ | ** {{done|Runs WebRTC tests}} | ||
** {{defer|Could also use this system to do a sanity check on the social API partner integration once the marionette based tests for social API are written}} | ** {{defer|Could also use this system to do a sanity check on the social API partner integration once the marionette based tests for social API are written}} | ||
** {{ | ** {{done|Should use an off-the-shelf scheduler - either jenkins or buildbot or something.}} | ||
=== P0: B2G General === | === P0: B2G General === | ||
* {{ | * {{done|Make B2G tests in TBPL more sheriffable, by providing a tool in TBPL (Treeherder?) to compare revisions of all B2G repos between arbitrary builds}} | ||
* {{done|Finish getting gaia-unit-tests and gaia-ui-tests running on all trees in TBPL}} | * {{done|Finish getting gaia-unit-tests and gaia-ui-tests running on all trees in TBPL}} | ||
* {{ | * {{miss|(Depends on James Lal's dev team): Deploy gaia-integration tests into TBPL}} | ||
** ''NOTE: We have an npm mirror | ** ''NOTE: We have an npm mirror; need to write mozharness script'' | ||
* {{ | * {{miss|Stand up tests on new Android 4.2 emulator (this is potentially going to be very time consuming)}} | ||
** ''NOTE: | ** ''NOTE: Waiting for rel-eng to schedule 4.2 builds/tests on cedar'' | ||
* {{done|Get b2g jsreftests working, so we can turn off no-ion builds/tests (stretch goal)}} | * {{done|Get b2g jsreftests working, so we can turn off no-ion builds/tests (stretch goal)}} | ||
** ''NOTE: We have these running in single-process mode, but reftest in general doesn't work in B2G in OOP mode, which is the reason for running them on no-ion builds, so we can't turn off the no-ion builds yet.'' | ** ''NOTE: We have these running in single-process mode, but reftest in general doesn't work in B2G in OOP mode, which is the reason for running them on no-ion builds, so we can't turn off the no-ion builds yet.'' | ||
* {{done|Get tests running using full-stack emulator builds}} | * {{done|Get tests running using full-stack emulator builds}} | ||
* {{defer|Implement heuristics to notify dev managers of "real" failures from B2G device test runs on Jenkins}} | * {{defer|Implement heuristics to notify dev managers of "real" failures from B2G device test runs on Jenkins}} | ||
* {{ | * {{miss|(Depends on rel-eng): Help implement a gaia try-server}} | ||
* {{ | * {{miss|Resolve various tap-related issues (including modal dialogs) in Marionette in order to support gaia-ui-tests and gaia-integration-tests}} | ||
=== P0: Eideticker === | === P0: Eideticker === | ||
* {{ | * {{done|Create responsiveness test for firefox on Android test using Eideticker}} | ||
* {{ | * {{done|Create a set (at least 2) of tests for b2g Eideticker to measure user perceived application startup and scrolling performance, and run on a daily basis}} | ||
=== P1: Marionette === | === P1: Marionette === | ||
* {{ | * {{done|Clean up oranges}} | ||
* {{ | * {{done|Better error reporting}} | ||
* {{drop|Add support for nested content processes}} | * {{drop|Add support for nested content processes}} | ||
** ''NOTE: not needed'' | ** ''NOTE: not needed'' | ||
* {{ | * {{done|Continue support for B2G testing / developer ergonomics (webqa, mach support) ... originally scheduled for Q2 but dropped for focus on gaia tests}} | ||
* {{drop|Support social API tests}} | * {{drop|Support social API tests}} | ||
** ''NOTE: The social API team is using the 'old' FirefoxDriver for now'' | ** ''NOTE: The social API team is using the 'old' FirefoxDriver for now'' | ||
| Line 59: | Line 59: | ||
** {{done|work with the games and js teams to run competitive benchmarks on android, b2g, and desktop platforms}} | ** {{done|work with the games and js teams to run competitive benchmarks on android, b2g, and desktop platforms}} | ||
** {{done|report numbers at least once a month}} | ** {{done|report numbers at least once a month}} | ||
** {{ | ** {{done|integrate additional benchmarks, browsers, platforms (i.e. IOS) as well as more hands off automation}} | ||
* Talos: | * Talos: | ||
** {{done|add additional alerts for disk flushes, too many disk accesses with talos xperf}} | ** {{done|add additional alerts for disk flushes, too many disk accesses with talos xperf}} | ||
** {{ | ** {{done|continue modifying existing tests to make the data more useful, improve documentation on what all tests measure}} | ||
** {{ | ** {{done|modify talos ts_dirty tests to have updated profiles and measure useful scenarios}} | ||
** {{ | ** {{miss|have sheriff/developer dashboards for datazilla, statistical analysis of all test types with alerts for detected regressions at <2% noise}} | ||
* Other: | * Other: | ||
** {{done|continue working with developers to refine microperf}} | ** {{done|continue working with developers to refine microperf}} | ||
** {{ | ** {{done|deliver a prototype for real world benchmark using the live internet to aid the js team}} | ||
** {{defer|provide help to the servo/rust team as they start measuring performance}} | ** {{defer|provide help to the servo/rust team as they start measuring performance}} | ||
** {{ | ** {{done|investigate ways to involve community into the process of measurement and analysis}} | ||
=== P1: MozBase === | === P1: MozBase === | ||
* {{ | * {{done|unittests for all modules}} | ||
* {{ | * {{miss|m-c commit watcher to notify mozbase owners of unexpected commits}} | ||
* {{ | * {{miss|full mochitest and reftest integration with mozprofile/mozprocess/mozrunner (excluding Fennec)}} | ||
** ''NOTE: mozprofile work done; the remainder is just being started'' | ** ''NOTE: mozprofile work done; the remainder is just being started'' | ||
* {{done|continue to standardize + impove dev practices}} | * {{done|continue to standardize + impove dev practices}} | ||
* {{done|structured logging}} | * {{done|structured logging}} | ||
* {{ | * {{done|manifestdestiny API improvements and fixes}} | ||
* {{done|add mozprocess documentation}} | * {{done|add mozprocess documentation}} | ||
=== P2: mozmill 2.0 released === | === P2: mozmill 2.0 released === | ||
* {{ | * {{done|Release 2.0 (should be done Q2)}} | ||
* {{done|SV leads metro testing using mozmill}} | * {{done|SV leads metro testing using mozmill}} | ||
| Line 91: | Line 91: | ||
Optimize bugzilla as a platform that we can build on top of. | Optimize bugzilla as a platform that we can build on top of. | ||
* {{done|Deploy native REST API}} | * {{done|Deploy native REST API}} | ||
* {{ | * {{done|Optimize and expand native REST API as needed to support 3rd party UI building efforts}} | ||
* {{ | * {{done|Stand up reviewboard as a semi-independent reviewing service}} | ||
* {{ | ** NOTE: Up at https://reviewboard.mozilla.org/, but waiting on proper cert and http -> https redirect from IT. | ||
* {{done|Tracking flag migration}} | |||
** Migration script is ready, but IT outage will not happen until early Q4. | |||
=== P0: TreeHerder === | === P0: TreeHerder === | ||
* {{ | * {{miss|Deliver UI parity with existing TBPL}} | ||
* {{ | * {{done|Implement UI plugin architecture}} | ||
* {{done|Set up publicly accessible instance}} | * {{done|Set up publicly accessible instance}} | ||
* {{done|Develop a logging strategy:}} | * {{done|Develop a logging strategy:}} | ||
| Line 106: | Line 108: | ||
=== P0: Datazilla === | === P0: Datazilla === | ||
* {{defer|Automate datazilla database rollover and rollover talos data. What do we want to do with legacy data? (Stretch goal)}} | * {{defer|Automate datazilla database rollover and rollover talos data. What do we want to do with legacy data? (Stretch goal)}} | ||
* {{ | * {{done|Refactor datazilla to remove all portions of the old UI, unnecessary controllers, and legacy code not in production}} | ||
* {{done|Provide datazilla landing page that displays data for all projects}} | * {{done|Provide datazilla landing page that displays data for all projects}} | ||
* {{ | * {{miss|Replace graph server for talos on m-c,m-i}} | ||
** Everything in place, except alerting which will carry into Q4 | |||
== Process == | == Process == | ||
=== P1: Infrastructure Load === | === P1: Infrastructure Load === | ||
* {{done|parallelize xpcshell test framework}} | * {{done|parallelize xpcshell test framework}} | ||
* {{ | * {{done|analyze always passing tests and determine a strategy for turning tests off selectively/running intermittently}} | ||
* {{ | * {{done|work with releng to turn on code coverage builds}} | ||
=== P1: Sheriff goals === | === P1: Sheriff goals === | ||
* {{ | * {{done|Provide documentation and how to steps to enable a community sheriff to be effective}} | ||
* {{ | * {{miss|Wire documentation into bugsAhoy/whatcanIdoForMozilla}} | ||
* {{ | * {{done|Fix P1/low hanging fruit of the Sheriffing Tools (see full goals list)}} | ||
* {{done|Drive treeherder UI functional specification}} | * {{done|Drive treeherder UI functional specification}} | ||
* [https://etherpad.mozilla.org/sheriffq3goals Full Goals here] | * [https://etherpad.mozilla.org/sheriffq3goals Full Goals here] | ||
| Line 125: | Line 128: | ||
=== P1: Bugmaster === | === P1: Bugmaster === | ||
* {{done|Create a contribution pipeline for 30 incoming bugmasters and track them through each stage of involvement in order to get 1 to the state where they regularly do triage.}} | * {{done|Create a contribution pipeline for 30 incoming bugmasters and track them through each stage of involvement in order to get 1 to the state where they regularly do triage.}} | ||
* {{ | * {{done|Improve Bugzilla Help pages.}} | ||
* [https://wiki.mozilla.org/Auto-tools/Projects/Bugmaster_2013Q3 Full goals here] | * [https://wiki.mozilla.org/Auto-tools/Projects/Bugmaster_2013Q3 Full goals here] | ||