Auto-tools/Goals/2014Q2: Difference between revisions
< Auto-tools | Goals
Jump to navigation
Jump to search
| (46 intermediate revisions by 7 users not shown) | |||
| Line 7: | Line 7: | ||
=== FxOS Certification Suite [owner: jgriffin] === | === FxOS Certification Suite [owner: jgriffin] === | ||
Support Mozilla's goal of expanding the uptake of FirefoxOS by OEM's by delivering the next version of the FxOS Certification Suite, which will contain the following: | Support Mozilla's goal of expanding the uptake of FirefoxOS by OEM's by delivering the next version of the FxOS Certification Suite, which will contain the following: | ||
* A guided (semi-auto) WebAPI test suite (to verify WebAPI functionality) | * {{done|}}A guided (semi-auto) WebAPI test suite (to verify WebAPI functionality) | ||
* The web-platform-tests (to cover HTML and DOM verification) | * {{done|}}The web-platform-tests (to cover HTML and DOM verification) | ||
* Static analysis tools (to verify changes to WebAPI IDL's and omni.ja) | * {{done|Static analysis tools (to verify changes to WebAPI IDL's and omni.ja)}} | ||
* A suite of tests to verify implementation of the permissions model | * {{done|}}A suite of tests to verify implementation of the permissions model | ||
* Related documentation | * {{done|}}Related documentation | ||
Tracking bug: {{bug|970615}}, etherpad: https://etherpad.mozilla.org/fxos-cert-suite | Tracking bug: {{bug|970615}}, etherpad: https://etherpad.mozilla.org/fxos-cert-suite | ||
| Line 17: | Line 17: | ||
=== MarionetteJS tests [owner: ahal] === | === MarionetteJS tests [owner: ahal] === | ||
Increase the impact and usefulness of the Gaia developer's integration tests, by working with them to enable these tests to run on devices and emulators. | Increase the impact and usefulness of the Gaia developer's integration tests, by working with them to enable these tests to run on devices and emulators. | ||
* Build a common harness that can run both Python and MarionetteJS tests. | * {{miss|}}Build a common harness that can run both Python and MarionetteJS tests. | ||
* Enable the harness to run on emulators and in on-device automation. | * {{miss|}}Enable the harness to run on emulators and in on-device automation. | ||
** On track for delivery in early Q3 | |||
== Performance == | == Performance == | ||
| Line 25: | Line 26: | ||
Improve our ability to make Talos test results more actionable by deploying a Talos Regression Dashboard. This dashboard will consume Talos alerts currently sent to dev.tree-management, identify regressions in the data, consolidate those regressions, and allow them to be "starred". In this quarter, we will: | Improve our ability to make Talos test results more actionable by deploying a Talos Regression Dashboard. This dashboard will consume Talos alerts currently sent to dev.tree-management, identify regressions in the data, consolidate those regressions, and allow them to be "starred". In this quarter, we will: | ||
* Deploy the dashboard to allow talos results to be "sheriffed" | * {{done|Deploy the dashboard to allow talos results to be "sheriffed"}} | ||
* Configure it to work additionally with regression alerts from datazilla (e.g., B2G alerts) | * {{miss|}}Configure it to work additionally with regression alerts from datazilla (e.g., B2G alerts) | ||
=== Eideticker [owner: wlach] === | === Eideticker [owner: wlach] === | ||
Get more value out of Eideticker, one of our more successful out-of-tree performance tools, by increasing its ability to scale: | Get more value out of Eideticker, one of our more successful out-of-tree performance tools, by increasing its ability to scale: | ||
* Make it easier to add new device types in B2G | * {{done|Make it easier to add new device types in B2G}} | ||
* Add test runs against the Tarako | * {{done|Add test runs against the Tarako device}} | ||
* Investigate ways to make it more cost efficient to scale for Android tests, by looking into external DeckLink boxes for HDMI capture | * {{done|Add test runs against the Flame devices}} | ||
* Run tests more frequently | * {{drop|}} Investigate ways to make it more cost efficient to scale for Android tests, by looking into external DeckLink boxes for HDMI capture | ||
* {{done|}} Run tests more frequently | |||
=== Datazilla [owner: jmaher] === | === Datazilla [owner: jmaher] === | ||
Improve our ability to monitor our products for performance regressions. | Improve our ability to monitor our products for performance regressions. | ||
* Minimal alerts for Talos regressions. | * {{done|}} Minimal alerts for Talos regressions. | ||
* Minimal alerts for mobile Eideticker regressions. | * {{done|}} Minimal alerts for mobile Eideticker regressions. | ||
** More complicated than originally thought, and Kyle's time being shared with charts. | |||
== Mobile == | == Mobile == | ||
| Line 47: | Line 50: | ||
Continue efforts to move our Android 2.3 testing to VM's and remove our reliance on tegras. | Continue efforts to move our Android 2.3 testing to VM's and remove our reliance on tegras. | ||
* Run reftests on Android 2.3 emulators on trunk. | * {{done|}} Run reftests on Android 2.3 emulators on trunk. | ||
* Designate Eideticker and/or Autophone as the official perf testing system for Android 2.3. | * {{done|}} Designate Eideticker and/or Autophone as the official perf testing system for Android 2.3. | ||
== Tools == | == Tools == | ||
| Line 55: | Line 58: | ||
Continue work on performance improvements; move off old systems to allow them to be decommissioned. | Continue work on performance improvements; move off old systems to allow them to be decommissioned. | ||
* Migrate BMO and upstream Bugzilla CI from Tinderbox to travis-ci. | * {{done|}} Migrate BMO and upstream Bugzilla CI from Tinderbox to travis-ci. | ||
* Continue and deepen analysis of BMO perf data. | * {{done|}} Continue and deepen analysis of BMO perf data. | ||
* | * {{done|}} Improve the slowest MySQL queries or include them in memcache. | ||
* {{done|}} Investigate and, if appropriate, implement improvements to page-load times, including CSS concatenation/minification and delayed loading of bug link tooltips. | |||
** Delayed-loading of bug-link tooltips was too complicated for the gains; going with a different solution (next quarter) that should give comparable benefits ({{bug|1028795}}) | |||
See [[BMO#Current_Projects]] for status. | See [[BMO#Current_Projects]] for status. | ||
| Line 63: | Line 68: | ||
=== Treeherder [owner: jeads] === | === Treeherder [owner: jeads] === | ||
Increase our ability to leverage existing and upcoming tools by deploying Treeherder, a TBPL successor. Treeherder will eventually allow us to have a single source of truth for test status, including on-device and extra-buildbot tests. In Q2: | Increase our ability to leverage existing and upcoming tools by deploying Treeherder, a TBPL successor. Treeherder will eventually allow us to have a single source of truth for test status, including on-device and extra-buildbot tests. In Q2: | ||
* Deploy a production version of Treeherder in coordination with Laura's team | * {{done|Deploy a production version of Treeherder in coordination with Laura's team}} | ||
* Define and implement a sheriff transition plan to move from TBPL to Treeherder | ** Carryover a socket.io using a different virtual host - that will be done in Q3 | ||
* Have Treeherder report data to OrangeFactor | * {{miss|Define and implement a sheriff transition plan to move from TBPL to Treeherder}} | ||
* Start planning on work needed to get performance data into Treeherder; this work will likely be fully delivered in Q3 | ** Treeeherder writes failure classifications to tbpl, and treeherder reports data to orange factor were what were needed to do this, but the transition plan was not implemented. Sheriffing transition will happen in early Q3. | ||
* {{done|Have Treeherder report data to OrangeFactor}} | |||
* {{done|Start planning on work needed to get performance data into Treeherder; this work will likely be fully delivered in Q3}} | |||
** Will go into Q3, but ahead of where we thought we'd be | |||
== Community == | == Community == | ||
| Line 72: | Line 80: | ||
=== Good First Bugs [owner: David Burns :automatedtester] === | === Good First Bugs [owner: David Burns :automatedtester] === | ||
Improve our ability to connect with contributors by making better use of the existing good first bugs list: | Improve our ability to connect with contributors by making better use of the existing good first bugs list: | ||
* Refresh this list on a monthly basis | * {{done|}} Refresh this list on a monthly basis | ||
* Review good first bugs and make sure they have adequate information for contributors to start with | * {{done|}} Review good first bugs and make sure they have adequate information for contributors to start with | ||
* Keep the numbers of open good first bugs steady or increasing | * {{done|}} Keep the numbers of open good first bugs steady or increasing | ||
= Supporting and Ongoing Tasks = | = Supporting and Ongoing Tasks = | ||
| Line 89: | Line 97: | ||
"keywords_type": "anywords", | "keywords_type": "anywords", | ||
"include_fields": "id, priority, summary, status, resolution, assigned_to, keywords", | "include_fields": "id, priority, summary, status, resolution, assigned_to, keywords", | ||
"order": "priority | "order": "priority" | ||
} | } | ||
</bugzilla> | </bugzilla> | ||
| Line 95: | Line 103: | ||
=== On-device tests === | === On-device tests === | ||
<bugzilla> | |||
{ | |||
"keywords": "ateam-b2g-device-big ateam-b2g-device-task", | |||
"keywords_type": "anywords", | |||
"include_fields": "id, priority, summary, status, resolution, assigned_to, keywords", | |||
"order": "priority DESC, bug_id" | |||
} | |||
</bugzilla> | |||
== Performance == | == Performance == | ||
| Line 102: | Line 116: | ||
=== B2G === | === B2G === | ||
<bugzilla> | |||
{ | |||
"keywords": "ateam-b2g-perf-big ateam-b2g-perf-task", | |||
"keywords_type": "anywords", | |||
"include_fields": "id, priority, summary, status, resolution, assigned_to, keywords", | |||
"order": "priority DESC, bug_id" | |||
} | |||
</bugzilla> | |||
=== Desktop === | === Desktop === | ||
| Line 121: | Line 140: | ||
=== Datazilla === | === Datazilla === | ||
<bugzilla> | |||
{ | |||
"keywords": "ateam-datazilla-big ateam-datazilla-goal ateam-datazilla-task", | |||
"keywords_type": "anywords", | |||
"include_fields": "id, priority, summary, status, resolution, assigned_to, keywords", | |||
"order": "priority DESC, bug_id" | |||
} | |||
</bugzilla> | |||
=== Talos === | === Talos === | ||
<bugzilla> | |||
{ | |||
"keywords": "ateam-talos-big ateam-talos-task", | |||
"keywords_type": "anywords", | |||
"include_fields": "id, priority, summary, status, resolution, assigned_to, keywords", | |||
"order": "priority DESC, bug_id" | |||
} | |||
</bugzilla> | |||
== Mobile == | == Mobile == | ||
=== Android === | === Android === | ||
* Green up more x86 tests and run them on trunk. | * Green up more x86 tests and run them on trunk - {{bug|936226}}. | ||
* Get some Android debug tests running on trunk. | * Get some Android debug tests running on trunk - {{bug|940068}}. | ||
== Tools == | == Tools == | ||
| Line 147: | Line 172: | ||
=== Marionette === | === Marionette === | ||
<bugzilla> | |||
{ | |||
"keywords": "ateam-marionette-big ateam-marionette-task ateam-marionette-goal", | |||
"keywords_type": "anywords", | |||
"include_fields": "id, priority, summary, status, resolution, assigned_to, keywords", | |||
"order": "priority DESC, bug_id" | |||
} | |||
</bugzilla> | |||
* Fix up major touch-related bugs [owner: mdas] | * Fix up major touch-related bugs [owner: mdas] | ||
| Line 161: | Line 195: | ||
* Create and execute on a plan to connect mozharness and mach [owner: ahal] | * Create and execute on a plan to connect mozharness and mach [owner: ahal] | ||
=== | === Unit tests === | ||
<bugzilla> | |||
{ | |||
"keywords": "ateam-unittests-big ateam-unittests-task ateam-unittests-goal", | |||
"keywords_type": "anywords", | |||
"include_fields": "id, priority, summary, status, resolution, assigned_to, keywords", | |||
"order": "priority DESC, bug_id" | |||
} | |||
</bugzilla> | |||
=== Sheriffing === | === Sheriffing === | ||
* More work on version-control-tools repo (eg bzexport, qimportbz, and other repo tools) | * More work on version-control-tools repo (eg bzexport, qimportbz, and other repo tools) | ||
= Backlog = | = Backlog = | ||
| Line 191: | Line 223: | ||
* Make B2G test jobs fail on uncaught JS exceptions [owner: ahal, {{bug|959493}}] | * Make B2G test jobs fail on uncaught JS exceptions [owner: ahal, {{bug|959493}}] | ||
* Add JSReftests to B2G [owner: ahal, {{bug|873179}}] | * Add JSReftests to B2G [owner: ahal, {{bug|873179}}] | ||
* Get WebAPI tests runnning on debug emulators [owner: ahal] | * Get WebAPI tests runnning on debug emulators [owner: ahal] | ||
* Get tests running on emulator-kk [owner: ahal] | * Get tests running on emulator-kk [owner: ahal] | ||
* Get tests running on Firefox Mulet, as soon as it's available [owner: ahal] | * Get tests running on Firefox Mulet, as soon as it's available [owner: ahal] | ||
* Auto-submit crash reports from automated tests [owner: TBD, {{bug|962700}}] | |||
== | == Tools == | ||
=== Unit tests === | |||
* WebDriver support in web-platform-tests | |||
* Create a tool to identify inter-test dependencies (GSOC project, Q2-Q3) [owner: jmaher] | |||
* Run valgrind on test slaves [{{bug|977240}}, blocked on RelEng making "real" valgrind builds with uploaded build artifacts - {{bug|977351}}] | * Run valgrind on test slaves [{{bug|977240}}, blocked on RelEng making "real" valgrind builds with uploaded build artifacts - {{bug|977351}}] | ||
== Performance == | |||
=== B2G === | |||
* Work with engineering to identify better ways to use gaia endurance and AWSY tests [owner: rwood] | |||
** Identify overlap between the two and turn off one if appropriate | |||
* Work with the FxOS_Perf team to stand up B2G power tests on the Flame [owner: TBD] | |||
=== Talos === | |||
* svg-opacity cleanup (revisit the test, possible rewrite/revision bump) | |||
* new scroll test (in addition to tscrollx) - work with vladan/avih | |||
Latest revision as of 19:17, 30 June 2014
Top Level Goals
A list of high priority goals we're committed to achieving in Q2.
B2G
FxOS Certification Suite [owner: jgriffin]
Support Mozilla's goal of expanding the uptake of FirefoxOS by OEM's by delivering the next version of the FxOS Certification Suite, which will contain the following:
- [DONE] A guided (semi-auto) WebAPI test suite (to verify WebAPI functionality)
- [DONE] The web-platform-tests (to cover HTML and DOM verification)
- [DONE] Static analysis tools (to verify changes to WebAPI IDL's and omni.ja)
- [DONE] A suite of tests to verify implementation of the permissions model
- [DONE] Related documentation
Tracking bug: bug 970615, etherpad: https://etherpad.mozilla.org/fxos-cert-suite
MarionetteJS tests [owner: ahal]
Increase the impact and usefulness of the Gaia developer's integration tests, by working with them to enable these tests to run on devices and emulators.
- [MISSED] Build a common harness that can run both Python and MarionetteJS tests.
- [MISSED] Enable the harness to run on emulators and in on-device automation.
- On track for delivery in early Q3
Performance
Talos Regression Dashboard [owner: jmaher]
Improve our ability to make Talos test results more actionable by deploying a Talos Regression Dashboard. This dashboard will consume Talos alerts currently sent to dev.tree-management, identify regressions in the data, consolidate those regressions, and allow them to be "starred". In this quarter, we will:
- [DONE] Deploy the dashboard to allow talos results to be "sheriffed"
- [MISSED] Configure it to work additionally with regression alerts from datazilla (e.g., B2G alerts)
Eideticker [owner: wlach]
Get more value out of Eideticker, one of our more successful out-of-tree performance tools, by increasing its ability to scale:
- [DONE] Make it easier to add new device types in B2G
- [DONE] Add test runs against the Tarako device
- [DONE] Add test runs against the Flame devices
- [DROPPED] Investigate ways to make it more cost efficient to scale for Android tests, by looking into external DeckLink boxes for HDMI capture
- [DONE] Run tests more frequently
Datazilla [owner: jmaher]
Improve our ability to monitor our products for performance regressions.
- [DONE] Minimal alerts for Talos regressions.
- [DONE] Minimal alerts for mobile Eideticker regressions.
- More complicated than originally thought, and Kyle's time being shared with charts.
Mobile
Android [owner: mcote]
Continue efforts to move our Android 2.3 testing to VM's and remove our reliance on tegras.
- [DONE] Run reftests on Android 2.3 emulators on trunk.
- [DONE] Designate Eideticker and/or Autophone as the official perf testing system for Android 2.3.
Tools
Bugzilla [owner: mcote]
Continue work on performance improvements; move off old systems to allow them to be decommissioned.
- [DONE] Migrate BMO and upstream Bugzilla CI from Tinderbox to travis-ci.
- [DONE] Continue and deepen analysis of BMO perf data.
- [DONE] Improve the slowest MySQL queries or include them in memcache.
- [DONE] Investigate and, if appropriate, implement improvements to page-load times, including CSS concatenation/minification and delayed loading of bug link tooltips.
- Delayed-loading of bug-link tooltips was too complicated for the gains; going with a different solution (next quarter) that should give comparable benefits (bug 1028795)
See BMO#Current_Projects for status.
Treeherder [owner: jeads]
Increase our ability to leverage existing and upcoming tools by deploying Treeherder, a TBPL successor. Treeherder will eventually allow us to have a single source of truth for test status, including on-device and extra-buildbot tests. In Q2:
- [DONE] Deploy a production version of Treeherder in coordination with Laura's team
- Carryover a socket.io using a different virtual host - that will be done in Q3
- [MISSED] Define and implement a sheriff transition plan to move from TBPL to Treeherder
- Treeeherder writes failure classifications to tbpl, and treeherder reports data to orange factor were what were needed to do this, but the transition plan was not implemented. Sheriffing transition will happen in early Q3.
- [DONE] Have Treeherder report data to OrangeFactor
- [DONE] Start planning on work needed to get performance data into Treeherder; this work will likely be fully delivered in Q3
- Will go into Q3, but ahead of where we thought we'd be
Community
Good First Bugs [owner: David Burns :automatedtester]
Improve our ability to connect with contributors by making better use of the existing good first bugs list:
- [DONE] Refresh this list on a monthly basis
- [DONE] Review good first bugs and make sure they have adequate information for contributors to start with
- [DONE] Keep the numbers of open good first bugs steady or increasing
Supporting and Ongoing Tasks
A list of ongoing and supporting tasks that the A*Team is actively engaged in. The delivery date for these will vary depending on the resource requirements of our Top Level goals, as well as the amount of support requests that come in during the quarter, and their respective priorities.
B2G
Tests in Continuous Integration
Bugzilla query error
Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 403 [1] => Forbidden ) ) 1
On-device tests
Bugzilla query error
Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 403 [1] => Forbidden ) ) 1
Performance
B2G
Bugzilla query error
Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 403 [1] => Forbidden ) ) 1
Desktop
- Implement desktop power tests, reporting to datazilla
- deploy rvitillio's power benchmark
- define set of websites to track
- on windows 7, automate browsing to the list of sites and recording power readings
- compare between aurora and central at least once a week, ideally nightly
- priority 2 is osx and linux
- mainthread I/O tracking
- work with aklotz to get this new feature tracked on an existing tbpl based job
- possible new benchmarks to be run nightly (not per commit) [p2]
- work with cpeterson to get additional benchmarks (think Tom's Hardware) which do not overlap with the games benchmarks
Datazilla
Bugzilla query error
Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 403 [1] => Forbidden ) ) 1
Talos
Bugzilla query error
Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 403 [1] => Forbidden ) ) 1
Mobile
Android
- Green up more x86 tests and run them on trunk - bug 936226.
- Get some Android debug tests running on trunk - bug 940068.
Tools
Bugzilla
- Improve Review Board Bugzilla extension to deepen integration.
- Extend history APIs to return all changes to bugs, instead of just bugs_activity entries, and allow the REST version to query a set of bugs instead of just one.
Marionette
Bugzilla query error
Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 403 [1] => Forbidden ) ) 1
- Fix up major touch-related bugs [owner: mdas]
- WebDriver Specification incompatibility bugs [owner: David Burns :AutomatedTester]
- Selenium Java and Python command executor interfaces [owner: ato]
- Running Selenium test suite on cedar [owner: ato]
- Add structured logging support [owner: ahal]
- Support Loop testing for dmose [owner: AutomatedTester]
Misc
- Release a new version of mozilla-build on Windows that uses the latest version of msys
- Help to stabilize the existing Steeplechase tests and expand the suite with new tests and capabilities
- Create and execute on a plan to connect mozharness and mach [owner: ahal]
Unit tests
Bugzilla query error
Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 403 [1] => Forbidden ) ) 1
Sheriffing
- More work on version-control-tools repo (eg bzexport, qimportbz, and other repo tools)
Backlog
A list of things we'd like to do, but are not actively working on, due to resource constraints, poor requirements, or outside dependencies.
Marionette
- Further WebDriver spec compatibility work [owner: ato] - Backlog
- Goal: Getting Marionette-backed FirefoxDriver deployed at major companies
B2G
- Set up a separate Jenkins instance for B2G so we can put the Jenkins scripts in version control
- Make B2G test jobs fail on uncaught JS exceptions [owner: ahal, bug 959493]
- Add JSReftests to B2G [owner: ahal, bug 873179]
- Get WebAPI tests runnning on debug emulators [owner: ahal]
- Get tests running on emulator-kk [owner: ahal]
- Get tests running on Firefox Mulet, as soon as it's available [owner: ahal]
- Auto-submit crash reports from automated tests [owner: TBD, bug 962700]
Tools
Unit tests
- WebDriver support in web-platform-tests
- Create a tool to identify inter-test dependencies (GSOC project, Q2-Q3) [owner: jmaher]
- Run valgrind on test slaves [bug 977240, blocked on RelEng making "real" valgrind builds with uploaded build artifacts - bug 977351]
Performance
B2G
- Work with engineering to identify better ways to use gaia endurance and AWSY tests [owner: rwood]
- Identify overlap between the two and turn off one if appropriate
- Work with the FxOS_Perf team to stand up B2G power tests on the Flame [owner: TBD]
Talos
- svg-opacity cleanup (revisit the test, possible rewrite/revision bump)
- new scroll test (in addition to tscrollx) - work with vladan/avih