QA/Test Automation/2010-11-17

previous meeting | Meetings | next meeting »

Dial in

 # 650-903-0800 or 650-215-1282 x92 Conf# 315 (US/INTL)
 # 1-800-707-2533 (pin 369) Conf# 315 (US)
 # irc.mozilla.org #mozmill for backchannel

Attendees

• Cameron, Geo, Al, Matt, Anthony, Aaron, Adrian, Henrik, Dave

Goals Overview

• QA Q4 goals

Project Status

Tests / Shared Modules

General (Henrik)

• Waiting for Clint's proposal how to do Drag&Drop - needed for Panorama tests

Broken Tests

• Firefox 4.0 (Geo)
• Firefox 3.5/3.6 (Anthony)
  • Landed waitForTabPanel() to make notification bar handling more reliable (5 of top-10 fails resolved)

DOM Walker + L10n API (Adrian)

• final review stages

Redesign Shared Modules (Geo)

Test Refactoring (Anthony)

• Draft styleguide being composed (targetting EOW)
• Begin refactoring once styleguide approved

Local Test Data (Anthony)

• No progress yet - considering community outreach to create testpages
  • 35 tests are already local
  • 4 tests remain which can be made local
  • remainder are not doable in Mozmill
  • not much value-add

Update Testing (Henrik)

• Currently we fail in downloading partial updates because the about window and the old ui are blocking each other. (bug 599290)
  • Attached is a workaround patch
• We will have to update the shared module asap to be able to test the new ui

Add-ons Testing (Henrik)

• With Mozmill 1.5.1 add-on authors should be able to create tests now. But should we wait with advertisement until shared modules have been refactored?
• Dave is working on tests for the Selenium IDE extension

Mozmill Crowd Extension (Henrik)

• In contact with Patrick Brunswig regarding IPC-PIPE regression. Will still use nsIProcess to prevent possible other upcoming regressions

Mozmill Result Dashboard (Henrik)

• Waiting to land patches for software update tests (multiple updates)

Automation / Infrastructure (Henrik)

• Investigating possible notification exchange between RelEng/QA for triggered test-runs

Personal Status

For the personal status please check the weekly status updates:

• 101115 - 101119

Roundtable

General

• Results from work week
• Endurance tests with Mozmill (see Etherpad document)
• MozMill testday on Nov 26th?
  • Yes, we want to have it
• Litmus vs. Mozmill tests
  • For the add-ons manager we have 60+ in-litmus requests. Do we want to have manual tests for all those requests or can we only implement those which really need manual action?
• Other
  • Etherpad
    • Any comments? What do you like? What don't you like? Any suggestions?
      • I can directly see the status of any team member without having to look at their status reports (Henrik)
      • I can see discussions even when I was offline (Henrik)
      • Etherpad makes it so easy and fast to edit content compared to the wiki (Henrik)

Issues

Meeting Notes

• Goals Review and Additions
  • [CARRY OVER] Mozmill Crowd extension
    • Test environments for Windows and Linux have to be setup
    • Need functionality to download and install the test environment
    • Update the extension to use nsIProcess instead of IPC-PIPE component
    • Demo of the extension during the all hands week

• • [ON TRACK] Mozmill Dashboard (update and add-on tests)
    • Once the automation script sends all the data up to brasstacks (later this week) we have to think about the visualization of the results
    • For add-on tests we will setup an addon testrun for Selenium IDE

• • [ON TRACK] Buildbot execution of tests with local test data
    • behind on creating new local test files - we can use the test day to get progress
    • 35 tests moved to local test data last quarter
    • 4 test files for this quarter / about 20 test files remaining
    • we should make sure that tests with already local test data are stable and can go to buildbot
    • what is the plan to get a specific set of tests to get run on buildbot
      • tag tests with meta data in the file which can be used by a script to copy over

• • [ON TRACK] Refactoring the shared module system
    • not a lot of progress in the work week
    • implementation of the initial classes
    • we will have at least the majority pieces in place to implement most of our tests
    • pick tests from nearly each of the existing test groups for a broad coverage

• • [ON TRACK] Mozmill Panorama tests
    • Lead needed for coordination -> Cameron? Who is 2nd?
    • Geo will talk with Clint about drag and drop requirements
    • Given by Anthony there should be 1/2 of the tests on Litmus be able to run without D&D
    • Geo will come up with some performance tests
    • Focus on already existent tests on Litmus
      • Figuring out with QA leads and devs which of those tests are important to automate

• • [NEW] Prototype for endurance tests
    • For first thoughts see the etherpad
    • Have to find a lead and 2nd

• • [NEW] Define trigger mechanism to start Mozmill test-runs
    • whatever sort of notifications even manual ones to start test-runs on all the platforms
    • Send out emails when failures happen
    • Big impact of Mozmill tests to run RelEng update tests
    • Talk to RelEng to define the type of notification (Lead: Geo, 2nd: Henrik)
      • No ETA for RelEng inclusion
      • We have requested a way to get notification from RelEng for our test-runs
      • Make our update tests absolutely trustful before running them in the RelEng environment (i.e nightly-test channel)

• Handling of manual testcases in Litmus for automated tests
  • Each automated test needs a manual test
  • Where they have to be located?
    • Keep in mind that Litmus 2 is not that far away.
    • jsdoc vs attachment on Bugzilla
      • jsdoc can handle changes more gracefully - one step with script changes
      • jsdoc comments can be automatically pulled
    • Everything should be on one place, mark tests for automation only

Action items

• ? Al (Anthony)?: Check existing tests if they are already covered by mochitests and worth to transform local
• Aaron (Geo): Tagging of tests and how to get them into buildbot
• [DONE] Geo (Henrik): Schedule meeting about trigger notifications for test-runs
• [DONE] Geo: Start thread about which parts the Etherpad document should cover
• [DONE] Matt (Henrik): Finding a lead and 2nd lead for the Panorama goal
• [DONE] Matt (Henrik): Goals review and addition of new goals to team goals
• [ON TRACK] Henrik: Work with Waverly on updating the spreadsheet for Add-ons Manager tests - add labels for automated tests