MozillaWiki

QA/Test Automation/2011-02-09: Difference between revisions

< QA‎ | Test Automation
 
(8 intermediate revisions by 3 users not shown)
Line 7: Line 7:


=Attendees=
=Attendees=
*
* Anthony, Dave, Henrik, Aaron, Al, Matt


= Last weeks action Items =
= Last weeks action Items =
Line 32: Line 32:
= Project Status =
= Project Status =


==On-demand update (bft) tests (Geo/Henrik)==
==[https://wiki.mozilla.org/QA/Mozmill_Test_Automation/On_Demand_Update_Testing On-demand update (bft) tests] (Geo/Henrik)==
* Script to stage builds for testing is in the work; WIP can already read the configuration file and download all referenced builds {{bug|628659}}
* Script to stage builds for testing is in the work; WIP can already read the configuration file and download all referenced builds {{bug|628659}}
* We preferred to rely on a Pulse implementation instead of triggers via ssh
* We preferred to rely on a Pulse implementation instead of triggers via ssh


==Endurance Tests (Dave/Anthony)==
==[https://wiki.mozilla.org/QA/Mozmill_Test_Automation/Endurance_Tests Endurance Tests] (Dave/Anthony)==
* Awaiting review from Henrik
* Awaiting review from Henrik
* Looking into getting the current test name from Mozmill
* Looking into getting the current test name from Mozmill
* Looking into an endurance test for [https://bugzilla.mozilla.org/show_bug.cgi?id=631494 bug 631494]
* Looking into an endurance test for [https://bugzilla.mozilla.org/show_bug.cgi?id=631494 bug 631494]


==Panorama Tests (Anthony/Al)==
==[https://wiki.mozilla.org/QA/Mozmill_Test_Automation/Panorama_Testing Panorama Tests] (Anthony/Al)==
* 9 / 21 tests now under development
* 9 / 21 tests now under development
* no new tests landed yet, but we are getting close
* no new tests landed yet, but we are getting close
Line 50: Line 50:
** wants to spend a year working on Mozmill and apply learning to other Mozilla projects
** wants to spend a year working on Mozmill and apply learning to other Mozilla projects


==Redesign Shared Modules (Geo/Henrik) ==
==[https://wiki.mozilla.org/QA/Mozmill_Test_Automation/Shared_Modules_Refactor Redesign Shared Modules] (Geo/Henrik) ==
* Geo started to work on sprints, Henrik will chime in this week
* Geo started to work on sprints, Henrik will chime in this week
* Documentation for several basic widgets has been started
* Documentation for several basic widgets has been started
Line 60: Line 60:


==Others==
==Others==
===Mozmill Crowd===
===[https://wiki.mozilla.org/QA/Mozmill_Test_Automation/Crowd_Testing Mozmill Crowd]===
* Henrik had a talk at FOSDEM ([http://slidesha.re/e2Sy66) slides])
* Henrik had a talk at FOSDEM ([http://slidesha.re/e2Sy66) slides])
* We got some great feedback and were able to confirm some unknown problems
* We got some great feedback and were able to confirm some unknown problems
* Change to non-blocking process execution is still in work - demo went fine
* Change to non-blocking process execution is still in work - demo went fine


===Dashboard===
===[https://wiki.mozilla.org/QA/Mozmill_Test_Automation/Dashboard Dashboard]===
* No updates
* No updates


=== Cloud Automation (Al/?) ===
=== Cloud Automation (Al/Henrik) ===
* This is blocked by needing IT to give us the server. This is {{bug|630426}}. Mrz and Phong are going back and forth on this.
* This is blocked by needing IT to give us the server. This is {{bug|630426}}. Mrz and Phong are going back and forth on this.
* Once we have a server, initial virtual machines will be put onto it.
* Once we have a server, initial virtual machines will be put onto it.
* Al is looking at the VMware command line tools for how we can manage VMs.
* Al is looking at the VMware command line tools for how we can manage VMs.


===Tests / Shared Modules===
===[https://wiki.mozilla.org/QA/Mozmill_Test_Automation/Test_Writing Tests] / [https://wiki.mozilla.org/QA/Mozmill_Test_Automation/Shared_Modules Shared Modules]===
; General (Henrik)
; General (Henrik)
* First Mozmill test by a developer ({{bug|631052}})
* First Mozmill test by a developer ({{bug|631052}})
Line 79: Line 79:
* Improvements for general tests when run in localized builds
* Improvements for general tests when run in localized builds


;Broken Tests
;[https://wiki.mozilla.org/QA/Mozmill_Test_Automation/Fixing_Broken_Tests Broken Tests]
* Firefox 4.0
* Firefox 4.0
** Down to 1-2 constant failures (some random orange)
** Down to 1-2 constant failures (some random orange)
Line 86: Line 86:
** 3.5: 13 failures (down from 14)
** 3.5: 13 failures (down from 14)


===Automation Scripts / Infrastructure (Henrik)===
===[https://wiki.mozilla.org/QA/Mozmill_Test_Automation/Automated_Testruns Automation Scripts] / Infrastructure (Henrik)===
* See on-demand update tests for the staging script
* See on-demand update tests for the staging script


Line 96: Line 96:
=Roundtable=
=Roundtable=
;General
;General
* WebQA Test Automation status inclusion?
* Simon Stewart post about webdriver inclusion into the Browser.
** [https://github.com/operasoftware/operadriver https://github.com/operasoftware/operadriver]
** [http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/webdriver http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/webdriver]


= Meeting Notes =
= Meeting Notes =
* Shadow Server
** Different vendors for SSL certificates
** Those certificates will contain whitelists for subdomains
* Cloud automation
** Blocked in some areas from IT (server setup)
** Research for initial plan is on Al's personal goals list
* Endurance
** Tests with add-ons or even other software (AV software) installed
** Adding other kinds of metrics
* Panorama
** Partly overlap with Mochitests is wanted
* Refactor Shared Modules
** Check if we can acquire community members to help us on the documentation; mention it on the project page
* Mozmill Crowd
** Still searching for the best location for test environments
* Web QA
** How do we want to include WebQA automation?
*** Renaming Test Automation -> Desktop Test Automation (2 separate teams)
*** Or adding WebQA projects to the Test Automation team page
*** There are interests to also talk about WebQA related stuff during our meeting


==Action items==
==Action items==
;Al
* Create project page for Cloud Testing project
* Find out if we can get metrics for the upcoming qatest vm
* Find out if it is possible for us to upload files to QMO (test environment)
;Henrik
* {{done|}} Add links to project pages for project headings
* Create PHP script for downloading test environments

Latest revision as of 17:08, 11 February 2011

previous meeting | Meetings | next meeting »

Dial in

 # 650-903-0800 or 650-215-1282 x92 Conf# 315 (US/INTL)
 # 1-800-707-2533 (pin 369) Conf# 315 (US)
 # irc.mozilla.org #mozmill for backchannel

Attendees

  • Anthony, Dave, Henrik, Aaron, Al, Matt

Last weeks action Items

Anthony
  • [CARRY OVER] Update refactoring discussion items and communicate to team
  • [DONE] Make sure spreadsheet is updated with Panorama status
Al
  • [DONE] Talk with Anthony to find the requirements for SSL certificates on the shadow server
Geo
  • Finish on-demand updates project page
  • Figure out what jetpack is doing for docgen (ask myk/analyze sdk)
  • Document failed test process from henrik's email at appropriate place
Henrik
  • [DONE] Send Geo desired place for failed test docs
  • [DONE] Add 32-bit version of Firefox to our daily testruns on Ubuntu 64-bit
Matt/Al
  • Matt/Al: work to find the goal for cloud automation goal

Goals Overview

Risky Goals

Project Status

On-demand update (bft) tests (Geo/Henrik)

  • Script to stage builds for testing is in the work; WIP can already read the configuration file and download all referenced builds bug 628659
  • We preferred to rely on a Pulse implementation instead of triggers via ssh

Endurance Tests (Dave/Anthony)

  • Awaiting review from Henrik
  • Looking into getting the current test name from Mozmill
  • Looking into an endurance test for bug 631494

Panorama Tests (Anthony/Al)

  • 9 / 21 tests now under development
  • no new tests landed yet, but we are getting close
  • New community contributor, adamFishHook
    • Undergrad at National University Singapore
    • found his way to Mozilla through researching open-source contribution
    • came on IRC, Aaron steered him toward Panorama
    • wants to spend a year working on Mozmill and apply learning to other Mozilla projects

Redesign Shared Modules (Geo/Henrik)

  • Geo started to work on sprints, Henrik will chime in this week
  • Documentation for several basic widgets has been started
  • jsdoc toolkit looks good so far except some minor issues with inheritance

Shadow test server (Al/Henrik)

  • This is bug 621794.
    • Al has pinged IT again for an update and an ETA for when this can go live. There has been no response on this since 2/2. We're gated on IT here.

Others

Mozmill Crowd

  • Henrik had a talk at FOSDEM (slides)
  • We got some great feedback and were able to confirm some unknown problems
  • Change to non-blocking process execution is still in work - demo went fine

Dashboard

  • No updates

Cloud Automation (Al/Henrik)

  • This is blocked by needing IT to give us the server. This is bug 630426. Mrz and Phong are going back and forth on this.
  • Once we have a server, initial virtual machines will be put onto it.
  • Al is looking at the VMware command line tools for how we can manage VMs.

Tests / Shared Modules

General (Henrik)
  • First Mozmill test by a developer (bug 631052)
  • Improvements for l10n tests
  • Improvements for general tests when run in localized builds
Broken Tests
  • Firefox 4.0
    • Down to 1-2 constant failures (some random orange)
  • Firefox 3.5/3.6 (Anthony/Aaron)
    • 3.6: 6 failures (down from 11)
    • 3.5: 13 failures (down from 14)

Automation Scripts / Infrastructure (Henrik)

  • See on-demand update tests for the staging script

Personal Status

For the personal status please check the weekly status updates:

Roundtable

General
  • WebQA Test Automation status inclusion?

Meeting Notes

  • Shadow Server
    • Different vendors for SSL certificates
    • Those certificates will contain whitelists for subdomains
  • Cloud automation
    • Blocked in some areas from IT (server setup)
    • Research for initial plan is on Al's personal goals list
  • Endurance
    • Tests with add-ons or even other software (AV software) installed
    • Adding other kinds of metrics
  • Panorama
    • Partly overlap with Mochitests is wanted
  • Refactor Shared Modules
    • Check if we can acquire community members to help us on the documentation; mention it on the project page
  • Mozmill Crowd
    • Still searching for the best location for test environments
  • Web QA
    • How do we want to include WebQA automation?
      • Renaming Test Automation -> Desktop Test Automation (2 separate teams)
      • Or adding WebQA projects to the Test Automation team page
      • There are interests to also talk about WebQA related stuff during our meeting

Action items

Al
  • Create project page for Cloud Testing project
  • Find out if we can get metrics for the upcoming qatest vm
  • Find out if it is possible for us to upload files to QMO (test environment)
Henrik
  • [DONE] Add links to project pages for project headings
  • Create PHP script for downloading test environments
Retrieved from "https://wiki.allizom.org/index.php?title=QA/Test_Automation/2011-02-09&oldid=284041"