QA/Test Automation/2010-12-01

previous meeting | Meetings | next meeting »

Dial in

 # 650-903-0800 or 650-215-1282 x92 Conf# 315 (US/INTL)
 # 1-800-707-2533 (pin 369) Conf# 315 (US)
 # irc.mozilla.org #mozmill for backchannel

Last weeks action Items

• ? Al (Anthony)?: Check existing tests if they are already covered by mochitests and worth to transform local
  • No progress
• Aaron (Geo): Tagging of tests and how to get them into buildbot
  • Can do a tag recognition script in about an hour or two if we're willing to accept bash+grep
  • Translating it into something more portable like Python will take extra time
  • Suggest letting me do the bash+grep version initially, then following up w/ Python
  • Open question: manifests or in-test tagging? Need to figure out ETA/effort estimate on manifest solution
• [DONE] Geo (Henrik): Schedule meeting about trigger notifications for test-runs
• [DONE] Geo: Start thread about which parts the Etherpad document should cover
• [DONE] Matt (Henrik): Finding a lead and 2nd lead for the Panorama goal
• [DONE] Matt (Henrik): Goals review and addition of new goals to team goals
• [ON TRACK] Henrik: Work with Waverly on updating the spreadsheet for Add-ons Manager tests - add labels for automated tests

Goals Overview

• QA Q4 goals

Project Status

Endurance Tests

• Hoping to kick this off today - Dave/Aaron
  • Kickoff ideas/brainstorming - notes

Panorama Tests

• 15 / 21 identified tests do not rely on Drag & Drop
• ashughes:
  • Spoke with Matt about taking lead on this and I am interested
  • I know Al and Cameron were interested as well so can we get a final word on this?

Triggered software updates (Geo)

• Note: If not already done by meeting, will be moving this into a project page.
• Proposed structure:
  1. Trigger comes from somewhere on build landing, contains current build id
     • Open question: Pulse vs. Email from Releng? In discussions now on both sides.
  2. Trigger polling script on VM catches trigger
  3. Current build downloaded and saved off for next trigger.
  4. Trigger polling script adds build to queue (directory full of semaphores or a flat file) for processing.
     • May not be needed for initial PoC as builds don't typically come frequently enough to queue. This would be an edge case handler.
  5. Queue polling script pulls the job off the queue, does two things:
     1. Downloads current build and saves it off for next trigger.
     2. Build saved off from last trigger will be updated and compared to expected build id.
  • Initial reporting to brasstacks, possiblly email on failures
  • Initial version for trunk, en_US only. Still targeting multi-platform.
• Things already there
  • Script to update a build and do some basic heuristics for testing exists.
• Things needed now
  • Update test script needs to be able to take a particular build id for verification
  • Trigger being decided upon between QA, RelEng, and Christian Legnitto (Pulse)
    • Eventually we want to be on Pulse, is unknown now whether it's sufficiently mature.
    • Go/no-go on Pulse to be made no later than next week, possibly as early as Friday.
    • Overall structure the same regardless.
  • Polling script
  • Queuing mechanism (? see above)
• Things needed eventually
  • RelEng wants to do this in their infrastructure eventually if at all possible.
  • If not possible, we'll need a way to report directly back to RelEng
  • Not there yet, though. For now want to run it on our side to make sure of robustness before putting in critical path.
• Complications
  • If we can't get per-platform notifications, may have to add what amounts to a multiplexer
    • Multiplexer would take one final notification and then notify all the VMs.
    • I'd rather not have to do this (it's complicated, delays testing, and possibly introduces deadlocks) so am seeking per-platform notifications.

Mozmill Crowd Extension (Henrik)

• Currently: Collecting dependencies and building test-environments for all platforms
• Next: move to nsIProcess API

Redesign Shared Modules (Geo)

• Slow progress to date largely due to priority juggling
• Plan of Action still the same:
  • Implement 80% case controls (button, textbox, menus/lists)
  • Implement control constructors that adapt DOMUtils and Elementlib methods of control binding
  • Implement 80% case methods (click, type, focus control, text verification)
  • Implement a sample test from as many subdirectories as possible for breadth.
  • 20% case controls/methods/etc to follow, but seeking max value up front.
• Class declaration is straightforward
• Map declaration less straightforward, attempts to date have been a little messy
• Timeboxing map declaration decision to next week
• After decision made, actual creation of map and conversion of tests is very quick.

Mozmill Result Dashboard (Henrik)

• Reworking project pages
• Next: Proposal for visualization and queries of update results

Build bot - local data (Anthony)

• No change - high resource cost, low value at this point
• Doable: 4 test pages + 1 test
• Much depends on proxy and certificate server handling which is not currently doable
• Not likely to see any change this quarter

Others

DOM Walker + L10n API (Adrian)

Tests / Shared Modules

• General (Henrik)
  • Refactored modal dialog has been landed on all branches
    • Some hidden older failures have been revealed
    • Some regressions introduced
    • Interests from Simon Steward to use the module for Selenium

• Test Refactoring (Anthony)
  • Styleguide drafted - final call for review/feedback
    NOTE: Please be aware that there are some things left out of this version of the styleguide. The initial version serves to communicate those things we have all already agreed on. Discussion and revision will improve on version 1.0.
    • Matt: What is the criteria of review comment vs review rejection? We should be very explicit on this so that there are no surprises with review results. NOTE: This might be more appropriate for this document and linking the two.
  • Tasks for Phase I have been outline and will go into a project page shortly - expect work to begin next week

• Broken Tests
  • Firefox 4.0 (Geo)
    • Dave Hunt also working on this
    • bug 592411 fixed, which fixed several topfails
    • Still continuing to pick tests off the Top Failures List and fixing as opportunity/time arises
  • Firefox 3.5/3.6 (Anthony)
    • 3.6: 18 Failures - up from 13 since modalDialogAPI
    • 3.5: 18 Failures - up from 9 since modalDialogAPI
    • Aaron focusing on modalDialogAPI refactoring in tests
    • Anthony focusing on pre-existing failures
    • Fix Rate: 3/week

Automation / Infrastructure

• Update Testing (Henrik)
  • Reports now support multiple updates in a row
• Add-ons Testing (Henrik)
  • no update

Personal Status

For the personal status please check the weekly status updates:

• 101129 - 101203

Roundtable

General

• Project management software (Henrik)
  • See weekly etherpad
  • Proposal for wiki pages

Issues