canmove, Confirmed users, Bureaucrats and Sysops emeriti
4,714
edits
QA/Test Automation/2010-11-17: Difference between revisions
→Action items
m (→Roundtable) |
|||
| (12 intermediate revisions by the same user not shown) | |||
| Line 7: | Line 7: | ||
=Attendees= | =Attendees= | ||
* Cameron, Geo, Al, Matt, Anthony, Aaron, Adrian, Henrik, Dave | |||
= Last weeks action Items = | |||
= Goals Overview = | |||
* [https://intranet.mozilla.org/QA/Q42010_QAgoals#Automation_QA_Goals QA Q4 goals] | |||
= Risky Goals = | |||
= Project Status = | |||
== Tests / Shared Modules == | |||
; General (Henrik) | |||
* Waiting for Clint's proposal how to do Drag&Drop - needed for Panorama tests | |||
; Broken Tests | |||
* Firefox 4.0 (Geo) | |||
* Firefox 3.5/3.6 (Anthony) | |||
** Landed waitForTabPanel() to make notification bar handling more reliable (5 of top-10 fails resolved) | |||
; DOM Walker + L10n API (Adrian) | |||
* final review stages | |||
; Redesign Shared Modules (Geo) | |||
; Test Refactoring (Anthony) | |||
* Draft styleguide being composed (targetting EOW) | |||
* Begin refactoring once styleguide approved | |||
; Local Test Data (Anthony) | |||
* No progress yet - considering community outreach to create testpages | |||
** 35 tests are already local | |||
** 4 tests remain which can be made local | |||
** remainder are not doable in Mozmill | |||
** not much value-add | |||
; Update Testing (Henrik) | |||
* Currently we fail in downloading partial updates because the about window and the old ui are blocking each other. (bug 599290) | |||
** Attached is a workaround patch | |||
* We will have to update the shared module asap to be able to test the new ui | |||
; Add-ons Testing (Henrik) | |||
* With Mozmill 1.5.1 add-on authors should be able to create tests now. But should we wait with advertisement until shared modules have been refactored? | |||
* Dave is working on tests for the Selenium IDE extension | |||
; Mozmill Crowd Extension (Henrik) | |||
* In contact with Patrick Brunswig regarding IPC-PIPE regression. Will still use nsIProcess to prevent possible other upcoming regressions | |||
; Mozmill Result Dashboard (Henrik) | |||
* | * Waiting to land patches for software update tests (multiple updates) | ||
; Automation / Infrastructure (Henrik) | |||
* Investigating possible notification exchange between RelEng/QA for triggered test-runs | |||
= | = Personal Status = | ||
For the | For the personal status please check the weekly status updates: | ||
* [http://etherpad.mozilla.com:9000/mozauto-101119 101115 - 101119] | |||
=Roundtable= | =Roundtable= | ||
;General | ;General | ||
* MozMill testday on Nov 26th? | * Results from [http://etherpad.mozilla.com:9000/33H4JCcOOv work week] | ||
** | * Endurance tests with Mozmill (see Etherpad document) | ||
* [http://etherpad.mozilla.com:9000/DJde7oLlEb MozMill testday] on Nov 26th? | |||
** Yes, we want to have it | |||
* Litmus vs. Mozmill tests | * Litmus vs. Mozmill tests | ||
** For the add-ons manager we have 60+ in-litmus requests. Do we want to have manual tests for all those requests or can we only implement those which really need manual action? | ** For the add-ons manager we have 60+ in-litmus requests. Do we want to have manual tests for all those requests or can we only implement those which really need manual action? | ||
| Line 33: | Line 75: | ||
** Etherpad | ** Etherpad | ||
*** Any comments? What do you like? What don't you like? Any suggestions? | *** Any comments? What do you like? What don't you like? Any suggestions? | ||
**** I can directly see the status of any team member without having to look at their status reports (Henrik) | |||
**** I can see discussions even when I was offline (Henrik) | |||
**** Etherpad makes it so easy and fast to edit content compared to the wiki (Henrik) | |||
;Issues | ;Issues | ||
=Meeting Notes= | = Meeting Notes = | ||
* Goals Review and Additions | |||
** {{prev|}} Mozmill Crowd extension | |||
*** Test environments for Windows and Linux have to be setup | |||
*** Need functionality to download and install the test environment | |||
*** Update the extension to use nsIProcess instead of IPC-PIPE component | |||
*** Demo of the extension during the all hands week | |||
** {{ok|}} Mozmill Dashboard (update and add-on tests) | |||
*** Once the automation script sends all the data up to brasstacks (later this week) we have to think about the visualization of the results | |||
*** For add-on tests we will setup an addon testrun for Selenium IDE | |||
** {{ok|}} Buildbot execution of tests with local test data | |||
*** behind on creating new local test files - we can use the test day to get progress | |||
*** 35 tests moved to local test data last quarter | |||
*** 4 test files for this quarter / about 20 test files remaining | |||
*** we should make sure that tests with already local test data are stable and can go to buildbot | |||
*** what is the plan to get a specific set of tests to get run on buildbot | |||
**** tag tests with meta data in the file which can be used by a script to copy over | |||
** {{ok|}} Refactoring the shared module system | |||
*** not a lot of progress in the work week | |||
*** implementation of the initial classes | |||
*** we will have at least the majority pieces in place to implement most of our tests | |||
*** pick tests from nearly each of the existing test groups for a broad coverage | |||
** {{ok|}} Mozmill Panorama tests | |||
*** Lead needed for coordination -> Cameron? Who is 2nd? | |||
*** Geo will talk with Clint about drag and drop requirements | |||
*** Given by Anthony there should be 1/2 of the tests on Litmus be able to run without D&D | |||
*** Geo will come up with some performance tests | |||
*** Focus on already existent tests on Litmus | |||
**** Figuring out with QA leads and devs which of those tests are important to automate | |||
** {{new|}} Prototype for endurance tests | |||
*** For first thoughts see the [http://etherpad.mozilla.com:9000/mozauto-101119 etherpad] | |||
*** Have to find a lead and 2nd | |||
** {{new|}} Define trigger mechanism to start Mozmill test-runs | |||
*** whatever sort of notifications even manual ones to start test-runs on all the platforms | |||
*** Send out emails when failures happen | |||
*** Big impact of Mozmill tests to run RelEng update tests | |||
*** Talk to RelEng to define the type of notification (Lead: Geo, 2nd: Henrik) | |||
**** No ETA for RelEng inclusion | |||
**** We have requested a way to get notification from RelEng for our test-runs | |||
**** Make our update tests absolutely trustful before running them in the RelEng environment (i.e nightly-test channel) | |||
* Handling of manual testcases in Litmus for automated tests | |||
** Each automated test needs a manual test | |||
** Where they have to be located? | |||
*** Keep in mind that Litmus 2 is not that far away. | |||
*** jsdoc vs attachment on Bugzilla | |||
**** jsdoc can handle changes more gracefully - one step with script changes | |||
**** jsdoc comments can be automatically pulled | |||
*** Everything should be on one place, mark tests for automation only | |||
==Action | ==Action items== | ||
* ? Al (Anthony)?: Check existing tests if they are already covered by mochitests and worth to transform local | |||
* Aaron (Geo): Tagging of tests and how to get them into buildbot | |||
* {{done|}} Geo (Henrik): Schedule meeting about trigger notifications for test-runs | |||
* {{done|}} Geo: Start thread about which parts the Etherpad document should cover | |||
* {{done|}} Matt (Henrik): Finding a lead and 2nd lead for the Panorama goal | |||
* {{done|}} Matt (Henrik): Goals review and addition of new goals to team goals | |||
* {{ok|}} Henrik: Work with Waverly on updating the spreadsheet for Add-ons Manager tests - add labels for automated tests | |||