Calendar:QA TestDays

Welcome to the Calendar Community Test Day!

Thank you for participating in this test day. The success of Sunbird and Lightning is heavily dependent on community members like you. We appreciate your support!

When you are done, please send feedback to the calendar mailing-list. We are very interested to hear about how the test day went, any overall thoughts about the Calendar project, and roll up of issues found and bugs filed.

The official time for the test day is Tuesday, August 22, 2006, from 12:00 UTC to 02:00 UTC, but anyone can jump on at any time on any day! Find out what time that is in your timezone

For more information about this release, please check out the Calendar Home Page. You will find the current release status, and more general information on the Calendar project.

Where do I start?

• Open up your favorite IRC client:
  • Windows clients
  • MacOS clients
  • Linux clients
  • ChatZilla (Firefox extension) -
• Connect to irc.mozilla.org
• Join the #calendar-qa channel

OK, I am in the channel, now what?

Let the moderators know if you have any questions or encounter any problems. If you are not familiar with our bugzilla bug tracking system, feel free to report your problems to a moderator and they can log any bugs for you.

Who will be moderating the #calendar-qa channel on the Test Days?

The moderator's nicks for this test day will be:

• ctalbert
• jminta
• lilmatt
• dmose

What build should I test?

We will be testing the Sunbird and Lightning nightly candidate builds:

Sunbird:

ftp://ftp.mozilla.org/pub/mozilla.org/calendar/sunbird/nightly/latest-trunk

Lightning (for Thunderbird 1.5.x and 2.0 candidates):

ftp://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/nightly/latest-mozilla1.8

Lightning (for Thunderbird 3.0 candidates):

ftp://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/nightly/latest-trunk (On Windows and Linux use the Thunderbird builds next to Lightning because of bug 321244, bug 343053)

What specifically will we be testing?

In a word, everything.

There are three main areas that we need to concentrate on. Here they are in order of importance.

Bugzilla Testing

Verify that the fixed bugs are addressed by these nightly builds.

Litmus Testing

There are four test suites in Litmus for Calendar. We want to run as many of these Litmus tests as possible for each product.

• Sunbird Basic Functional Tests - basic tests for Sunbird
• Sunbird Full Functional Tests - in depth tests for Sunbird
• Lightning Basic Functional Tests - basic tests for Lightning
• Lightning Full Functional Tests - in depth tests for Lightning

Usability Testing

From the release overview and the Calendar Charter, we have several high level goals that we want the 0.3 Calendar release to achieve. We need your help and creativity in ensuring that the nightly candidates for 0.3 move the Calendar project toward meeting these goals.

Testing Instructions

Follow these instructions:

1. Download Sunbird or Lightning from the above URLs

THEN - Verify Bugzilla Fixed Bugs

1. Click on the above Bugzilla bug list link and verify Fixed bugs. Read how the bug was produced and try to recreate it to verify the error doesn't happen anymore.
2. If the error is no longer occuring, select the "Mark bug as Verified" radio button and click the Commit button. If the error still occurs, select Reopen Bug and enter comments as to why the bug is being reopened (i.e. what is still failing), then click the Commit button.

OR - Run Litmus Test Cases

1. Click on the above Litmus test case link and run test cases. Follow the test steps and verify the function under test works as described in Expected Results.
2. If the test case performs as expected, mark it "Passed". If not, mark it as "Failed" and then enter a bug (see below).

OR - Perform Usability Testing

1. Use Sunbird and Lightning like you normally would. Try to do everything an end-user might do, ranging from simple calendar operations to power user features. Try to think of actions that are not typical since you will most likely find a bug there.

BUG FOUND!

1. When you think you've found a bug, try to reproduce it. Restart Sunbird/Lightning and do the same steps again. The most valuable bugs are those that can be reproduced.
2. Once the bug has been reproduced, search for it or something similar on Bugzilla. This can easily be done by searching for the error message, window name, or some over identifing feature of the bug. If you found an existing bug that is just like the bug you found, read the comments. Maybe your situation is slightly different and your input could help developers find the origin of the bug.
3. If you can't find the bug in Bugzilla, talk about it in the channel. Maybe others can confirm your bug or find a similar bug in bugzilla. This also gives moderators a chance to guide you to either provide more details, update an existing bug or file a new bug.
4. If you are certain you found a unique bug and have read the Bug Writing Guidelines and the Bugzilla Etiquette go ahead and file the bug in Bugzilla. Remember to be as specific as possible.

Some other ways you can help:

• Think of some test cases that aren't already included in Litmus (we love suggestions!).
• Ask your friends to join you. The more people help the better!

Where do I find Litmus and Bugzilla?

• Litmus can be found at http://litmus.mozilla.org/
• A list of Calendar specific test cases can be found via the search function

• Bugzilla can be found at http://bugzilla.mozilla.org/
• A list of Calendar specific bugs (without enhancements) can be found here
• A list of already filed enhancement requests can be found here

Tips:

• Make sure you enter the correct information for your platform, version of Thunderbird, Lightning etc. Please include the full build identifier (i.e Thunderbird 1.5.0.5 (20060719) with Lightning 0.1+ (2006080907)).
• Be sure to read the Testing Instructions above before filing a bug.
• Feel free to ask on IRC if you need help or have any more questions.
• TODO: Put in the Build ID for sunbird since people can't just find it.
• When running tests in Litmus, wait to submit all the tests as one batch (use the submit button at the bottom of the test list).
• When running tests in Litmus, click on the test name to go to the next test, not "View Test".
• TODO: come up with an idea on how to achieve the most test coverage. Perhaps we can talk to people about what tests they'd like to run and suggest certain areas. Or perhaps we can encourage them to do ad-hoc testing. We could have some kind of goal regarding the "Personal Testing Coverage" and the "Community Testing Coverage" metrics from the Litmus run tests page. Other thoughts?

How do I send feedback?

Please use email us at dev-apps-calendar at lists.mozilla.org to send feedback about how the testday went, any overall thoughts about Sunbird and Lightning, and to follow up on issues found and bugs filed. We really appreciate your help today and your feedback is very valuable.

Rewards!

There will be two prizes given. Both prizes will be $25 gift certificates to the Mozilla Store. There are two categories for winners.

• Prize #1 goes to the person that completes the most tests. This means that they verify the most Bugzilla fixes and/or complete the most Litmus test cases.
• Prize #2 goes to the person that finds the best set of bugs. These bugs will be high-profile ones as deemed by the calendar QA team and the 0.3 release drivers as high profile, possibly blocking bugs.
• As always, know that we appreciate everyone's contribution and you all have our heart felt gratitude. Thank you.