Auto-tools/Goals/2014Q3

Top Level Goals

A list of high priority goals we're committed to achieving in Q3.

B2G

Per-commit on-device tests for B2G [owner: mdas]

[DROPPED] (moved to B2G team) Get on-device tests running a small set of perf and functional tests per-commit (joint with WebQA) on Flames - bug 1019792

• Adapt mozpool to B2G - bug 1020366
• Report on-device functional test results to Treeherder
• Get gaia-integration tests running on-device with full support for handling crashes, timeouts, and test manifests
• Identify a small set of performance and correctness tests that we can run every commit (of gaia or gecko+gaia) (joint with qa - jsmith and nhirata will help with defining *what* should run in the sanity test)
• Get crash reporting working on gaia-ui-tests and b2gperf tests running on-device

FxOS Branding Certification Suite [owner: jgriffin]

[DONE] Continue to iterate on the FxOS Cert Suite with additional features and support.

• [DONE] Release a B2G 1.4-compatible version of the suite.
• [DONE] Continue to expand our semi-automatic WebAPI test suite.
• [DONE] Investigate adding support for non-phone devices.
• [DONE] Release a B2G 2.0-compatible version of the suite.
• [DROPPED] Stand up automated tests of the cert suite to help prevent regressions.
  • Development of the suite has slowed down and made this not as important.

TaskCluster [owner: gbrown]

[DONE] Investigate and fix problems running B2G tests in the TaskCluster environment.

Developer Productivity

ReviewBoard [owner: mcote]

• [MISSED] Deploy ReviewBoard for developers to start using (joint with RelEng)
  • The lead developer underwent emergency surgery and was out for 4 weeks. We are 99% of the way there, with only one fix to go before launch.

Performance

Talos and Regression Alerts [owner: jmaher]

• [DONE] New talos tests for tp5o scroll, media tests (webRTC), mainthread IO in tp5, and webGL
• [DONE] Datazilla alerts in beta mode (full parity with graph server alerts) and reduced noise [ekyle/jmaher]
  • [DONE] per page and per suite alerts
  • [DONE] reduced noise so we can file bugs with same percentage of fixed/wontfix as we currently get with graph server alerts
  • [DONE] use datazilla alerts as primary source of finding regressions (not depending on graph server alerts)
    • Will continue to iterate on improvements in Q4.

Eideticker [owner: wlach]

• [DONE] Run B2G tests more frequently across multiple branches, the same as b2gperf runs on
• [DONE] Get Android 2.3 and 4.x runs working again with increased frequency

Games Benchmarking [owner: dminor]

• [DONE] Games Benchmarking: get the webaudio test running against Firefox and Chrome

Treeherder [owner: jeads]

• [DONE] Deliver performance web service for ingesting and returning performance data
• [MISSED] Deliver UI for Talos data
  • Probably going to miss the performance UI in Q3 in order to fully hit the transition to Treeherder goal. We plan to get performance UI done in Q4 if that happens.
  • Result Set Performance UI
    • Add ability to toggle a performance view for individual pushes with charts of replicates
    • Add ability to store failure classifications for performance data

Sheriffing [owner: dburns]

• [DONE] Fully transition sheriffing away from TBPL (newsgroup post).

General Automation

• [DONE] Create weekly reports that describe how many tests have been added/disabled/enabled per suite and platform [ahal/akachkach]
• [DONE] Move reftest to mozbase [ted]

Marionette [owner: David Burns :AutomatedTester]

• [DONE] Add command executor for Python; this means finishing Marionette transport layers for other languages and shipping them - Github

Bugzilla [owner: mcote]

• Performance
  • [DONE] Improve load time of related bugs; can decrease show_bug load times by up to 12% [glob]
    • bug 651803's response time improved from 10.6 seconds down to 8.6 seconds
  • [DONE] Minify and concatenate JS files [glob]
    • each http connection carries significant overheads https://developers.google.com/speed/docs/best-practices/rtt#CombineExternalJS
    • currently loading show_bug results in 21 js requests of 425k
    • 12 files are not minified (the others are minified yui2 js)
    • while the files are loaded in parallel, the "connection" timing varies between 500ms and more than 2 seconds
    • possible savings for a cold load of show_bug could be up to 2 seconds
    • warm loads of show_bug will probably not be impacted due to caching of responses
• Dev productivity
  • [DONE] Authoritative view for review history [dylan]
  • [DONE] Rewrite REST API docs into something comprehensible [dkl]

Community [owner: jmaher, dburns]

Our main problem is keeping the contributors we have and converting them to longer term contributors. So we need to define a set of skill levels and a path for contribution - better stepping stones.

• [DONE] Create good_next_bugs (name can be adjusted) so once contributors are comfortable they can do more serious coding/problem solving on a project they are familiar with.
• [DONE] Monthly review of mentored bugs and projects
  • ensure we have well documented good first bugs (GFB) and plenty of them per project
  • ensure projects page is not far out of date and all GFB's have a related project page
  • ensure we have a good queue of mentored but not GFB's (i.e. good_next_bugs)
  • ensure all mentored bugs have valid mentors and are still valid bugs

Supporting and Ongoing Tasks

A list of ongoing and supporting tasks that the A*Team is actively engaged in. The delivery date for these will vary depending on the resource requirements of our Top Level goals, as well as the amount of support requests that come in during the quarter, and their respective priorities.

B2G

• [DROPPED] (moved to B2G team) Refactor gaia-endurance tests
  • make them take full memory reports
  • have them report to AWSY when run on emulators
  • review tests with developers

• [DROPPED] (moved to B2G team) Gu on emulator
• [DONE] Gu fewer dependencies on external resources (like browser tests) [bug 1027621 - davehunt]
• [DONE] Get tests running against Mulet in production
• [DONE] cppunit tests on B2G emulators - Bug 1014653 [dminor]

Performance

• [DONE] Talos tests - put webgl tests into production [jmaher]
• [DONE] power profiling tests [jmaher]
• [DONE] Add Eideticker for Android to Jenkins CI: bug 1015382 [wlach]
• Move Eideticker to a real database instead of JSON files [wlach]
• [DONE] Address b2g usability issues in Datazilla from the list in bug 913726 [jeads]

Dashboards [owner: ekyle]

The following are dashboard requests we've received. Some of these may turn into work items for new or existing dashboards.

• Add ability to track velocity of sprints over time (smooney)
• Break down milestones and programs in order to isolate complications before they impact timelines (sescalante)
• Add ability to track crash and regression keywords (kairo)
• Add ability to investigate reopened bugs (lhenry)
• Track team workload over time (cpeterson)
• Track the age of partner bugs (nfong)
• Track fuzzer bugs in order to assess the effectiveness of fuzzing strategies over time (abillings)
• Sprint status dashboard, like old b2g but more gates (elancaster)
• Aggregate all that has been built and summarize what is useful (kyle)
• Track enabled/disabled tests over time (ahal)

Android [owner: mcote]

• Investigate try-like server for autophone
• Make autophone developer friendly for local use
• Make autophone report run status to Treeherder

General Automation

• [DONE] Green up tests on Windows 64-builds [chmanchester]
• Convert xpcshell to Structured Logging [chmanchester]
• Implement new security model for Pulse [mcote]
• [DEFER] Implement mochitest-chrome for Android and B2G (? - may drop)
  • Will do this in Q4.

Sheriffs

• [DROPPED] Help to define the Windows environment used in TaskCluster [ryanvm]

Supporting Events

Primary Team	Action	Grade
Platform Team	Find Owners for top oranges from Orangefactor	B (Was shown in platform meetings but owners were reluctant)
Gecko/B2G/Gaia	Checkin-needed patches are landed within 48 Hours	A
Gecko/B2G/Gaia/Release Management	Uplifts patches are landed within 48 Hours	[DROPPED]

Marionette

• [DROPPED] Get WebDriver spec tests running on TBPL - Depends on Top level bug
  • Testharness/TestRunner for webdriver integrated with WPTRunner
  • Test tests on Cedar and green accordingly

• [DROPPED] Complete test writing for some subsection of the WD spec
  • Upstream our tests for the WD spec. (marionette tests that spec the WD spec)

• Feature complete dburns
  • Modal/ Dialog support
  • Fennec support
  • window management
  • [DROPPED] log (If it lands in the spec)

web-platform-tests [owner: jgraham]

• [DONE] Green up tests on one (or more) platform(s) and schedule on trunks branches
• Resolve intermittent issues on Mac OS X
• Improve and test WebDriver framework (written by Marc Fisher from Google)
• [DROPPED] WebDriver test executor for wptrunner - needs a little more definition

Bugzilla [owner: mcote]

• [DEFER] Make page load times for enter bug faster
• [DEFER] Create a plan to improve search UX to help people do faster, more relevant searches
• [DONE] New: We identified that JSON serialization improvements will help performance more than the above.

Community [owner: jgriffin]

• Create badges for three different levels of contribution, see: http://people.mozilla.org/~dhunt/automation_badges/final/