View source for User:Rcampbell/3PM Testing Meeting notes

TODO: Ask if Alice is available to help out with performance testing on tinderboxes.

TODO: Connecting Tests to Tinderboxes

Talk to Rob Helmer, Paul Reid re: build and Tinderboxen.

Graydon has new page set for incorporation into performance tests.

=== Planning Meeting ===

[http://people.mozilla.com/~timr/QATeam/Slide0001.gif 6 month schedule]

=== Unit Test Framework ===

Proof of Concept - 1 wk

Roll out to Trunk +1 week

Integrate with test Tinderboxes

Database + Reporting Mechanism

Integrate with Bob Clary's JavaScript testing systems

make file targets for unit tests. Developers should run locally to be ensure unit tests pass before committing code.

=== Test Harnesses for Acceptance Test Frameworks ===

DOM

Layout

Graphical Comparisons

Eggplant incorporation

JSUnit for testing content scripting

Jssh for testing into the chrome

XPCShell might have overlap with UnitTest

Test cases go to Litmus or Tinderbox?

"Underneath the GUI"

=== Performance Testing / Memory Leaks ===

Alice to help setup/improve performance tests on Tinderbox.

=== The Path to Automation ===

==== Top level datapoints ====

Eggplant

JavaScript Test Integration

Security Tests - non public, exposure issue. Recovery issue.

Unit Tests

Acceptance Tests - Top Site testing

Performance / Memory Leak tests

L10N

Update Testing

==== Reporting ====

Eggplant is currently running on an old machine, needs updating. Has Perl::Litmus to dump data into Litmus.

JS Tests are already executable via command line tool. Need to massage data to unified format for storage/reporting. Coop thinks Litmus would be the place to store the results and needs to talk to Bob to make the storage decisions. Perl::Litmus can be used for this as well.

Issue with deciding how to generate or create tests in Litmus.

Security Tests could be automated with EggPlant. Litmus could have a "private" tag added to hide tests.

Unit Tests could be run through Tinderbox and pass some more detailed results to Litmus. Requires further investigation.

Acceptance Tests run through Tinderbox with results fed into Litmus.

Performance / Memory Leaks run through Tinderbox. Possibly need more well-formed results format from different tests for feeding into multiple systems. Needs further discussion with Timr, Alice and Schrep.

L10N talk with Rob Helmer about Tinderbox integration. Eggplant for verification. Dave, Tracy and Axel need to coordinate for Firefox 2 release.

Update Testing. Scary area. Require further analysis to break down failure points and study the different permutations.