Auto-tools/Goals/2014Q2

Top Level Goals

A list of high priority goals we're committed to achieving in Q2.

B2G

FxOS Certification Suite [owner: jgriffin]

Support Mozilla's goal of expanding the uptake of FirefoxOS by OEM's by delivering the next version of the FxOS Certification Suite, which will contain the following:

• A guided (semi-auto) WebAPI test suite (to verify WebAPI functionality)
• The web-platform-tests (to cover HTML and DOM verification)
• Static analysis tools (to verify changes to WebAPI IDL's and omni.ja)
• A suite of tests to verify implementation of the permissions model
• Related documentation

Tracking bug: bug 970615, etherpad: https://etherpad.mozilla.org/fxos-cert-suite

Performance

Talos Regression Dashboard [owner: jmaher]

Improve our ability to make Talos test results more actionable by deploying a Talos Regression Dashboard. This dashboard will consume Talos alerts currently sent to dev.tree-management, identify regressions in the data, consolidate those regressions, and allow them to be "starred". In this quarter, we will:

• Deploy the dashboard to allow talos results to be "sheriffed"
• Configure it to work additionally with regression alerts from datazilla (e.g., B2G alerts)

Eideticker [owner: wlach]

Get more value out of Eideticker, one of our more successful out-of-tree performance tools, by increasing its ability to scale:

• Make it easier to add new device types in B2G
• Add test runs against the Tarako and Flame devices
• Investigate ways to make it more cost efficient to scale for Android tests, by looking into external DeckLink boxes for HDMI capture
• Run tests more frequently

Mobile

TBD, define a path off of the tegras?

Tools

Bugzilla [owner: mcote]

TBD, performance? reviewboard?

Treeherder [owner: jeads]

Increase our ability to leverage existing and upcoming tools by deploying Treeherder, a TBPL successor. Treeherder will eventually allow us to have a single source of truth for test status, including on-device and extra-buildbot tests. In Q2:

• Deploy a production version of Treeherder in coordination with Laura's team
• Define and implement a sheriff transition plan to move from TBPL to Treeherder
• Have Treeherder report data to OrangeFactor
• Start planning on work needed to get performance data into Treeherder; this work will likely be fully delivered in Q3

Community

Good First Bugs [owner: David Burns :automatedtester]

Improve our ability to connect with contributors by making better use of the existing good first bugs list:

• Refresh this list on a monthly basis
• Review good first bugs and make sure they have adequate information for contributors to start with
• Keep the numbers of open good first bugs steady or increasing

Supporting and Ongoing Tasks

A list of ongoing and supporting tasks that the A*Team is actively engaged in. The delivery date for these will vary depending on the resource requirements of our Top Level goals, as well as the amount of support requests that come in during the quarter, and their respective priorities.

B2G

Tests in TBPL

• Get tests running on emulator-kk [owner: ahal]
• Get tests running on Firefox Mulet, as soon as it's available [owner: ahal]
• Get gaia-ui-tests running on emulators in TBPL [owner: davehunt, bug 916368]
• Help get gaia-integration tests running on emulators in TBPL [owner: TBD]
• Resolve cases where we aren't getting crash stacks from B2G automation [owner: ahal]

On-device tests

• Help get gaia-integration tests running on-device [bug 979077]
• Support the Taipei team by handling bugs and feature requests related to Mean Time Between Failure tests (MTBF) [owner: mdas]
• Support WebQA by handling bugs and feature requests related to gaia-ui-tests [owner: mdas]
• Auto-submit crash reports from automated tests [owner: TBD, bug 962700]

Performance

B2G

• Work with engineering to identify better ways to use gaia endurance and AWSY tests [owner: rwood]
  • Identify overlap between the two and turn off one if appropriate
• Work with the FxOS_Perf team to stand up B2G power tests on the Flame [owner: TBD]

Desktop

• Implement desktop power tests, reporting to datazilla

Datazilla

• Persist the date range in the UI [owner: jeads]
• Allow ordering by gecko or gaia commits dates; will allow us to display data from bisection runs [owner: jeads]
• Continue work to fine-tune datazilla ingestion rate and performance regression alerts [owner: kyle]

Talos

• Latency tests
• WebRTC tests
• Session restore tests
• SPS profiling

Tools

Marionette

• Fix up major touch-related bugs [owner: mdas]
• WebDriver Specification incompatibility bugs [owner: David Burns :AutomatedTester]
• Selenium Java and Python command executor interfaces [owner: ato]
  • Running Selenium test suite on cedar [owner: ato]
• Further WebDriver spec compatibility work [owner: ato]
  • Goal: Getting Marionette-backed FirefoxDriver deployed at major companies
• Add structured logging support [owner: ahal]
• Support Loop testing for dmose [owner: AutomatedTester]

Misc

• Release a new version of mozilla-build on Windows that uses the latest version of msys
• Help to stabilize the existing Steeplechase tests and expand the suite with new tests and capabilities

Mochitests

• Add structured logging support (GSOC project, Q2-Q3) [owner: jgraham]
• Create a tool to identify inter-test dependencies (GSOC project, Q2-Q3) [owner: jmaher]
• Create and execute on a plan to connect mozharness and mach [owner: ahal]

Sheriffing

• More work on version-control-tools repo (eg bzexport, qimportbz, and other repo tools)

Web-platform-tests

• WebDriver support in web-platform-tests
• continued work on getting web-platform-tests on TBPL

Backlog

A list of things we'd like to do, but are not actively working on, due to resource constraints or outside dependencies.

Marionette

• Further WebDriver spec compatibility work [owner: ato] - Backlog
  • Goal: Getting Marionette-backed FirefoxDriver deployed at major companies

B2G

• Set up a separate Jenkins instance for B2G so we can put the Jenkins scripts in version control
• Make B2G test jobs fail on uncaught JS exceptions [owner: ahal, bug 959493]
• Add JSReftests to B2G [owner: ahal, bug 873179]
• Automatically submit crash reports from on-device automation
• Get WebAPI tests runnning on debug emulators [owner: ahal]

Desktop

• Run valgrind on test slaves [bug 977240, blocked on RelEng making "real" valgrind builds with uploaded build artifacts - bug 977351]