canmove, Confirmed users, Bureaucrats and Sysops emeriti
4,714
edits
Auto-tools/Goals/2013Q4: Difference between revisions
→Desktop QA Automation (whimboo)
| (15 intermediate revisions by 7 users not shown) | |||
| Line 13: | Line 13: | ||
| Run endurance tests without Marionette issues | | Run endurance tests without Marionette issues | ||
| Run a set of endurance tests without gaia actions (or with simplified gaia actions to test only marionette) for 3 days w/o issues | | Run a set of endurance tests without gaia actions (or with simplified gaia actions to test only marionette) for 3 days w/o issues | ||
| {{ | | {{done| }} | ||
| Mdas | | Mdas | ||
| {{bug|908261}} has been marked a dupe, what will help here is {{bug|931044}} which we could hook up to run for 3 days | | {{bug|908261}} has been marked a dupe, what will help here is {{bug|931044}} which we could hook up to run for 3 days | ||
| Line 62: | Line 62: | ||
| Continue Bugzilla as a Platform transition | | Continue Bugzilla as a Platform transition | ||
| Deploy BzAPI compatibility layer, Change notification POC, Memcache minimum viable product by end of quarter | | Deploy BzAPI compatibility layer, Change notification POC, Memcache minimum viable product by end of quarter | ||
| {{done| }} | | {{done| }} (except bzAPI compatibility layer, eta early Q1) | ||
| Glob | | Glob | ||
| {{bug|885464}} | | {{bug|885464}} | ||
| Line 75: | Line 75: | ||
|- | |- | ||
| Ensure steeplechase webRTC automation as hands off as possible | | Ensure steeplechase webRTC automation as hands off as possible | ||
| | | Deployed mochitests, ability to change machine configuration to system by end of quarter. C++ tests were not created (not our responsibility) so did not get integrated. | ||
| {{ | | {{done| }} | ||
| Ted | | Ted | ||
| {{bug|894559}} | | {{bug|894559}} | ||
| Line 89: | Line 89: | ||
|- | |- | ||
| Deploy Mozmill 2.0, Metro infrastructure and tests, and aid SoftVision to take over day to day operations for QA automation with our oversight | | Deploy Mozmill 2.0, Metro infrastructure and tests, and aid SoftVision to take over day to day operations for QA automation with our oversight | ||
| | | Deployed projects, tried to get the SV team to take over the reigns, and we had marginal success. Had to bail them out a few times. | ||
| {{ | | {{done| }} | ||
| Whimboo | | Whimboo | ||
| {{bug|922200}} | | {{bug|922200}} | ||
| Line 107: | Line 107: | ||
=== Marionette (mdas) === | === Marionette (mdas) === | ||
* P0 - Run set of endurance tests for 3 days without marionette issues | * {{done|}} P0 - Run set of endurance tests for 3 days without marionette issues | ||
** space issues, memory usage, import log, and don't regress socket.timeout | ** space issues, memory usage, import log, and don't regress socket.timeout | ||
| Line 130: | Line 130: | ||
* {{miss|}} P0 [wlach] Deliver a suite of Eideticker responsiveness tests for FirefoxOS | * {{miss|}} P0 [wlach] Deliver a suite of Eideticker responsiveness tests for FirefoxOS | ||
* {{drop|}} P1 [wlach supporting] (crossteam)(with gaia team) github autolanding | * {{drop|}} P1 [wlach supporting] (crossteam)(with gaia team) github autolanding | ||
* P1 [ctalbert coordinating] expanding hardware rig in mtn view - (stephend team, IT) joint with perf system | * {{done|}} P1 [ctalbert coordinating] expanding hardware rig in mtn view - (stephend team, IT) joint with perf system | ||
{| class="wikitable collapsible" | {| class="wikitable collapsible" | ||
| Line 152: | Line 152: | ||
* {{miss|}} Q4 top priority is to have minimal set of documentation (what the tests do, how they are run, what they measure) and usage statistics (number of test runs, results) of our tools (all a-team involved performance tools/projects) available from a single place (scaled down dashboard) | * {{miss|}} Q4 top priority is to have minimal set of documentation (what the tests do, how they are run, what they measure) and usage statistics (number of test runs, results) of our tools (all a-team involved performance tools/projects) available from a single place (scaled down dashboard) | ||
* P0 [jmaher] Complete investigation into Talos suites with desktop performance team | * {{done|}} P0 [jmaher] Complete investigation into Talos suites with desktop performance team | ||
** tsvg, tscroll, ts, tpaint are all refreshed/updated accordingly. | |||
* {{miss|}} P0 [jmaher, ekyle] Complete alerting analysis for datazilla data | * {{miss|}} P0 [jmaher, ekyle] Complete alerting analysis for datazilla data | ||
* {{done|}} P0 [jmaher] hold a monthly cross team performance meeting to discuss tests, future projects, problems, relative priority | * {{done|}} P0 [jmaher] hold a monthly cross team performance meeting to discuss tests, future projects, problems, relative priority | ||
| Line 200: | Line 201: | ||
* Continue Bugzilla as a Platform (BaaP): | * Continue Bugzilla as a Platform (BaaP): | ||
** {{done|}} [mcote] Proof of concept of the Bugzilla Change Notification System (with security approval) | ** {{done|}} [mcote] Proof of concept of the Bugzilla Change Notification System (with security approval) | ||
** [dkl] (crossteam) bzAPI compatibility layer | ** {{miss|}} [dkl] (crossteam) bzAPI compatibility layer | ||
** [dkl] plan for versioning APIs, first draft of interface for API v2 | ** {{miss|}} [dkl] plan for versioning APIs, first draft of interface for API v2 | ||
* [glob] (crossteam) memcached minimum viable product (also helps BaaP above) | *** We had some interruptions due to more API work, developer tools, and antispam efforts. | ||
* {{done|}} [glob] (crossteam) memcached minimum viable product (also helps BaaP above) | |||
{| class="wikitable collapsible" | {| class="wikitable collapsible" | ||
| Line 208: | Line 210: | ||
|- | |- | ||
| | | | ||
* [mcote] ReviewBoard integration plan based on feedback | * {{defer|}} [mcote] ReviewBoard integration plan based on feedback | ||
* [mcote] somehow get over to git, or at least have a really serious plan for it | ** Discussion is still ongoing as to whether Review Board can meet our needs or not. | ||
* [kyle] deploy new bzETL | * {{done|}} [mcote] somehow get over to git, or at least have a really serious plan for it | ||
** Plan in place. | |||
* {{done|}} [kyle] deploy new bzETL | |||
|} | |} | ||
| Line 216: | Line 220: | ||
* {{done|}} Q4 top priority is remote reftest profiling in support of running reftests in reasonable time on Android 4.0/Panda and Android 4.2/x86. | * {{done|}} Q4 top priority is remote reftest profiling in support of running reftests in reasonable time on Android 4.0/Panda and Android 4.2/x86. | ||
* mach commands for running remote tests | * {{miss|}} mach commands for running remote tests | ||
** automatically get xpcshell/xre for remote tests | ** automatically get xpcshell/xre for remote tests | ||
** [dminor] Make the tooling around running gdb/jimdb a bit friendlier (mach command)? | ** [dminor] Make the tooling around running gdb/jimdb a bit friendlier (mach command)? | ||
| Line 227: | Line 231: | ||
* Continue working with releng to increase panda stability | * Continue working with releng to increase panda stability | ||
* [bc] Use Autophone to do one-off cross browser bandwidth metrics | * [bc] Use Autophone to do one-off cross browser bandwidth metrics | ||
* Deploy C++ remote tests | * {{done|}} [dminor] Deploy C++ remote tests | ||
|} | |} | ||
=== WebRTC (ted) === | === WebRTC (ted) === | ||
* {{drop|}} P0: Get Steeplechase into releng staging (joint with releng) | * {{drop|}} P0: Get Steeplechase into releng staging (joint with releng) | ||
* P0: Deploy C++ tests for SteepleChase | * {{miss|}} P0: Deploy C++ tests for SteepleChase | ||
* P0: Deploy ability to change machine configuration per test | * {{done|}} P0: Deploy ability to change machine configuration per test | ||
* P1: Investigate hardware testing set up (joint with desktop QA) | * {{defer|}} P1: Investigate hardware testing set up (joint with desktop QA) | ||
=== Bugmaster (lizzard) === | === Bugmaster (lizzard) === | ||
* Deploy bugmaster community building tools (bug board, tutorial, and bugzilla user profile report) | * {{miss|Deploy}} bugmaster community building tools (bug board, tutorial, and bugzilla user profile report) | ||
* Use summit to build out community involvement | * {{done|Use}} summit to build out community involvement | ||
* Continue triaging and imrpove the groundswell. | * {{done|Continue}} triaging and imrpove the groundswell. | ||
* Research into new Bugzilla home page for logged-in/logged-out users (separate pages) | * {{miss|Research}} into new Bugzilla home page for logged-in/logged-out users (separate pages) | ||
=== Desktop QA Automation (whimboo) === | === Desktop QA Automation (whimboo) === | ||
* (crossteam) Get SoftVision to lead the automation efforts for QA with our oversight | * {{done|(crossteam) Get}} SoftVision to lead the automation efforts for QA with our oversight | ||
* (crossteam) Switch Mozmill CI from Mozmill 1.5.x to Mozmill 2.0 | * {{done|(crossteam) Switch}} Mozmill CI from Mozmill 1.5.x to Mozmill 2.0 | ||
* (crossteam) Get Firefox Metro tests executed in production | * {{miss|(crossteam) Get}} Firefox Metro tests executed in production | ||
{| class="wikitable collapsible" | {| class="wikitable collapsible" | ||
| Line 251: | Line 255: | ||
|- | |- | ||
| | | | ||
* Extend Mozmill CI to cover most of our localized beta and release candidate builds | * {{miss|}} Extend Mozmill CI to cover most of our localized beta and release candidate builds | ||
* Create web application to generate the beta and release Jenkins job configuations | * {{done|}} Create web application to generate the beta and release Jenkins job configuations | ||
* (stretched) Setup tryserver Mozmill CI system assist team members in testing their patches | * (stretched) Setup tryserver Mozmill CI system assist team members in testing their patches | ||
* (stretched) Investigate requirements for community driving tool by working together with jdm, lizzard, and pieros | * (stretched) Investigate requirements for community driving tool by working together with jdm, lizzard, and pieros | ||