Auto-tools/Goals/2014Q3

Top Level Goals

A list of high priority goals we're committed to achieving in Q2.

B2G

Per-commit on-device tests for B2G [owner: mdas]

Get on-device tests running a small set of perf and functional tests per-commit (joint with WebQA) on Flames - bug 1019792

• Adapt mozpool to B2G - bug 1020366 [bc]
• Report on-device functional test results to Treeherder [owner TBD]
• Get gaia-integration tests running on-device with full support for handling crashes, timeouts, and test manifests [ahal]
• Identify a small set of performance and correctness tests that we can run every commit (of gaia or gecko+gaia) (joint with qa - jsmith and nhirata will help with defining *what* should run in the sanity test)
• Get crash reporting working on gaia-ui-tests and b2gperf tests running on-device [davehunt]

FxOS Branding Certification Suite [owner: jgriffin]

Continue to iterate on the FxOS Cert Suite with additional features and support.

• Release a B2G 1.4-compatible version of the suite.
• Continue to expand our semi-automatic WebAPI test suite.
• Stand up automated tests of the cert suite to help prevent regressions.
• Investigate adding support for non-phone devices.

TaskCluster [owner: gbrown]

Investigate and fix problems running B2G tests in the TaskCluster environment.

Developer Productivity

ReviewBoard [owner: mcote]

• Deploy ReviewBoard for developers to start using (joint with RelEng)

Performance

Talos and Regression Alerts [owner: jmaher]

• New talos tests for tp5o scroll, media tests (webRTC), mainthread IO in tp5, and webGL
• Datazilla alerts in beta mode (full parity with graph server alerts) and reduced noise [ekyle/jmaher]
  • per page and per suite alerts
  • reduced noise so we can file bugs with same percentage of fixed/wontfix as we currently get with graph server alerts
  • use datazilla alerts as primary source of finding regressions (not depending on graph server alerts)

Eideticker [owner: wlach]

• Run B2G Eideticker tests more frequently across multiple branches, the same as b2gperf runs on
• Get Android tests running with camera with a 2.3 and 4.x phone

Games Benchmarking [owner: dminor]

• Games Benchmarking: get the webaudio test running against Firefox and Chrome

Treeherder [owner: jeads]

• Deliver performance web service for ingesting and returning performance data
• Deliver UI for Talos data
  • Result Set Performance UI
    • Add ability to toggle a performance view for individual pushes with charts of replicates
    • Add ability to store failure classifications for performance data
  • Performance Summary Tab
    • See a top-level overview of performance data per branch, platform, etc (similar to the Datazilla UI)

Sheriffing [owner: dburns]

• Fully transition sheriffing away from TBPL

General Automation

• Create weekly reports that describe how many tests have been added/disabled/enabled per suite and platform [akachkach]
• Move reftest to mozbase [ted]

Marionette [owner: ato]

• Add command executors for Java and Python; this means finishing Marionette transport layers for other languages and shipping them

Bugzilla [owner: mcote]

• Performance
  • Improve load time of related bugs; can decrease show_bug load times by up to 12% [glob]
  • Minify and concatenate JS files [glob]
    • each http connection carries significant overheads https://developers.google.com/speed/docs/best-practices/rtt#CombineExternalJS
    • currently loading show_bug results in 21 js requests of 425k
    • 12 files are not minified (the others are minified yui2 js)
    • while the files are loaded in parallel, the "connection" timing varies between 500ms and more than 2 seconds
    • possible savings for a cold load of show_bug could be up to 2 seconds
    • warm loads of show_bug will probably not be impacted due to caching of responses
• Dev productivity
  • Authoritative view for review history [dylan]
  • Rewrite REST API docs into something comprehensible [dkl]

Community [owner: jmaher, dburns]

Our main problem is keeping the contributors we have and converting them to longer term contributors. So we need to define a set of skill levels and a path for contribution - better stepping stones.

• Create good_next_bugs (name can be adjusted) so once contributors are comfortable they can do more serious coding/problem solving on a project they are familiar with.
• Monthly review of mentored bugs and projects
  • ensure we have well documented good first bugs (GFB) and plenty of them per project
  • ensure projects page is not far out of date and all GFB's have a related project page
  • ensure we have a good queue of mentored but not GFB's (i.e. good_next_bugs)
  • ensure all mentored bugs have valid mentors and are still valid bugs

Supporting and Ongoing Tasks

A list of ongoing and supporting tasks that the A*Team is actively engaged in. The delivery date for these will vary depending on the resource requirements of our Top Level goals, as well as the amount of support requests that come in during the quarter, and their respective priorities.

B2G

• Refactor gaia-endurance tests [rwood]
  • make them take full memory reports
  • have them report to AWSY when run on emulators
  • review tests with developers

• Gu on emulator [jgriffin/davehunt]
• Gu fewer dependencies on external resources (like browser tests) [davehunt]
• Get tests running against Mulet in production [ahal]
• cppunit tests on B2G emulators - Bug 1014653 [dminor]
  • started, will probably not finish until Q3

Performance

• Talos tests - webgl, new svg/animation tests Ensure the scope is constrained on this one [jmaher]
• power profiling tests [jmaher]
• Add Eideticker for Android to Jenkins CI: bug 1015382 [bc or davehunt]
• Move Eideticker to a real database instead of JSON files [wlach]
• Address b2g usability issues in Datazilla from the list in bug 913726 [jeads]

Dashboards [owner: ekyle]

The following are dashboard requests we've received. Some of these may turn into work items for new or existing dashboards.

• Add ability to track velocity of sprints over time (smooney)
• Break down milestones and programs in order to isolate complications before they impact timelines (sescalante)
• Add ability to track crash and regression keywords (kairo)
• Add ability to investigate reopened bugs (lhenry)
• Track team workload over time (cpeterson)
• Track the age of partner bugs (nfong)
• Track fuzzer bugs in order to assess the effectiveness of fuzzing strategies over time (abillings)
• Aggregate all that has been built and summarize what is useful (kyle)

Android [owner: mcote]

• Investigate try-like server for autophone
• Make autophone developer friendly for local use
• Make autophone report run status to Treeherder

General Automation

• Green up tests on Windows 64-builds [chmanchester]
• Convert xpcshell to Structured Logging [chmanchester]
• Implement new security model for Pulse [mcote]
• Implement mochitest-chrome for Android and B2G (? - may drop)

Sheriffs

• Help to define the Windows environment used in TaskCluster [ryanvm]

Supporting Events

Primary Team	Action	Grade
Platform Team	Find Owners for top oranges from Orangefactor
Gecko/B2G/Gaia	Checkin-needed patches are landed within 48 Hours
Gecko/B2G/Gaia/Release Management	Uplifts patches are landed within 48 Hours

Marionette

• Get WebDriver spec tests running on TBPL - Depends on Top level bug
  • Testharness/TestRunner for webdriver integrated with WPTRunner
  • Test tests on Cedar and green accordingly

• Complete test writing for some subsection of the WD spec
  • Upstream our tests for the WD spec. (marionette tests that spec the WD spec)

• Feature complete dburns
  • Modal/ Dialog support
  • Fennec support
  • window management
  • log (If it lands in the spec)

web-platform-tests [owner: jgraham]

• Green up tests on one (or more) platform(s) and schedule on trunks branches
• Resolve intermittent issues on Mac OS X
• Improve and test WebDriver framework (written by Marc Fisher from Google)
• WebDriver test executor for wptrunner - needs a little more definition

Bugzilla [owner: mcote]

• Make page load times for enter bug faster
• Create a plan to improve search UX to help people do faster, more relevant searches

Community [owner: jgriffin]

• Create badges for three different levels of contribution, see: http://people.mozilla.org/~dhunt/automation_badges/final/