Calendar:QA TestDays: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
No edit summary
Line 5: Line 5:
'''Previous TestDays'''
'''Previous TestDays'''
*[[Calendar:QA_TestDay:2006-08-22|Aug 22, 2006]]
*[[Calendar:QA_TestDay:2006-08-22|Aug 22, 2006]]
<font size=6 color=red>
= '''RESULTS!''' =
* 661 Tests Ran in Litmus!
* 16 New Bugs!
* 3 new Blocking Defects Discovered!
</font>
Many thanks to everyone involved. I'd like to thank several people who went above and beyond the call to help out: Sebo confirmed bugs, helped to ensure one of our blockers got noticed, and put together a CalDav server for testing. xFallenAngel took the lead with Litmus and sorted out the inconsistancies in our testcases. He also set up a WebDav server for the test day. Johannes worked with him to sort out the Litmus test case issues. Thanks also to our channel moderators, dmose, lilmatt, and jminta. I appreciate everyone's help, and I hope to see you all at our next Calendar Test Day Event.
The Stars:
{| border=2 cellpadding=2
|-
| Johannes || lilmatt || jminta
|-
| Andreas || atrogu || cpm
|-
| demo-n || LisaH || mschroder
|-
| rogerk || sebo || SirBlackheart
|-
|Vali_Systm || VenomousGecko || Vamplr3
|-
| Taleisin || xFallenAngel || LonelyBob
|-
| karnesky || ahz || jeizer_a
|-
| tbodner || excessory || Damian
|-
| bbbrowning || torusturtle || celina63
|-
|}
In all, the Test Day was a resounding success. I look forward to doing this again with everyone. We will start having regular test days in as we near the 0.3 release date.
Remember that two rewards were given for the day. One reward for the most tests run/bugs verified. And one for the best bug found. With that in mind, the winners are....
*'''Karnesky''' takes the first prize with a whopping '''278''' tests run in Litmus!
*'''Sebo''' takes the second prize for confirming two blocking defects, setting up calendar servers, and working hard to make the test day a success.
= Welcome to the Calendar Community Test Day! =
Thank you for participating in this test day. The success of Sunbird and Lightning is heavily dependent on community members like you.  We appreciate your support!
When you are done, please send feedback to [mailto:dev-apps-calendar@lists.mozilla.org the calendar mailing-list].  We are very interested to hear about how the test day went, any overall thoughts about the Calendar project, and roll up of issues found and bugs filed.
The official time for the test day is '''Tuesday, August 22, 2006, from 12:00 UTC to 02:00 UTC''', but anyone can jump on at any time on any day!
[http://www.timeanddate.com/worldclock/fixedtime.html?month=8&day=22&year=2006&hour=12&min=00&sec=0&p1=0 Find out what time that is in your timezone]'''
''For more information about this release, please check out the [[Calendar:Home_Page| Calendar Home Page]].  You will find the current release status, and more general information on the Calendar project.''
== Where do I start? ==
* Open up your [http://www.ircreviews.org/clients/ favorite IRC client]:
** [http://www.ircreviews.org/clients/platforms-windows.html Windows clients]
** [http://www.ircreviews.org/clients/platforms-macos.html MacOS clients]
** [http://www.ircreviews.org/clients/platforms-unix.html Linux clients]
** [https://addons.mozilla.org/firefox/16/ ChatZilla (Firefox extension)] -
* Connect to [irc://irc.mozilla.org irc.mozilla.org]
* Join the #calendar-qa channel
== OK, I am in the channel, now what? ==
Let the moderators know if you have any questions or encounter any problems.  If you are not familiar with our [https://bugzilla.mozilla.org/index.cgi bugzilla bug tracking system], feel free to report your problems to a moderator and they can log any bugs for you.
== Who will be moderating the #calendar-qa channel on the Test Days? ==
The moderator's nicks for this test day will be:
* ctalbert
* jminta
* lilmatt
* dmose
== What build should I test? ==
We will be testing the Sunbird and Lightning nightly candidate builds:
'''Sunbird:'''
: ftp://ftp.mozilla.org/pub/mozilla.org/calendar/sunbird/nightly/latest-trunk
'''Lightning''' (for Thunderbird 1.5.x and 2.0 candidates):
: ftp://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/nightly/latest-mozilla1.8
'''Lightning''' (for Thunderbird 3.0 candidates):
: ftp://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/nightly/latest-trunk (On Windows and Linux use the Thunderbird builds next to Lightning because of {{bug|321244}}, {{bug|343053}})
== What specifically will we be testing? ==
In a word, everything.
There are three main areas that we need to concentrate on. Here they are in order of importance.
=====Bugzilla Testing=====
Verify that the [https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&short_desc_type=allwordssubstr&short_desc=&product=Calendar&component=Alarms&component=Calendar+Views&component=General&component=Import+and+Export&component=Internal+Components&component=Printing&component=Provider%3A+CalDav&component=Provider%3A+ICS%2FWebdav&component=Provider%3A+Local+Storage&component=Provider%3A+WCAP&component=Security&component=Sunbird+Only&component=Tasks&component=Website&long_desc_type=allwordssubstr&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&keywords_type=allwords&keywords=&bug_status=RESOLVED&resolution=FIXED&emailtype1=substring&email1=&emailtype2=substring&email2=&bugidtype=include&bug_id=&votes=&chfieldfrom=-1m&chfieldto=Now&chfield=resolution&chfieldvalue=&cmdtype=doit&order=Reuse+same+sort+as+last+time&field0-0-0=noop&type0-0-0=noop&value0-0-0= fixed bugs] are addressed by these nightly builds.
=====Litmus Testing=====
There are four test suites in [http://litmus.mozilla.org Litmus] for Calendar. We want to run as many of these Litmus tests as possible for each product.
* Sunbird Basic Functional Tests - basic tests for Sunbird
* Sunbird Full Functional Tests - in depth tests for Sunbird
* Lightning Basic Functional Tests - basic tests for Lightning
* Lightning Full Functional Tests - in depth tests for Lightning
=====Usability Testing=====
From the release [[Calendar:0.3_Release_Status|overview]] and the [[Calendar:Charter|Calendar Charter]], we have several high level goals that we want the 0.3 Calendar release to achieve. We need your help and creativity in ensuring that the nightly candidates for 0.3 move the Calendar project toward meeting these goals.
<font color=red>
== BEFORE YOU START--BACKUP YOUR PROFILE ==
</font>
Once you have downloaded the version of the application you want to test, but '''before''' you start it, you MUST back up your profile and create a testing profile, otherwise, you run a very good chance of losing your data.
Backing up a profile is easy:<br>
1. Find your profile directory:
{| border="2" cellpadding="2"
! OS !! Thunderbird Profile !! Sunbird Profile
|-
| Windows || %AppData%\Thunderbird || %AppData%\Mozilla\Sunbird
|-
| Linux || ~/.thunderbird || ~/.mozilla/sunbird
|-
| MacOSX || ~/Library/Thunderbird || ~/Library/Application Support/Sunbird
|}
2. Copy the root of it somewhere else, for example copy the Thunderbird or Sunbird directory and everything below it.
Creating a testing profile is also easy:
# Launch your application with a -P option
# Create a New Profile by clicking the Create a New Profile button
# Click Next, Give it a name like "Testing Profile", click Finish
# Select the testing profile and click the "Start Calendar" (Or Start Mail in Thunderbird) button.
'''Remember that you will have to start the application with the -P option during testing so you can pick your Testing profile'''
Once you've finished testing, you can delete the extra profiles and you'll be back to normal.
If something goes terribly wrong (although we don't expect it to), close the application and just replace the profile with the backed up copy. That will reset the application to the way it was before you started testing.
== Testing Instructions ==
# Download Sunbird or Lightning from the above URLs
# Create your testing profile (see above)
'''THEN - Verify Bugzilla Fixed Bugs'''
# Click on the above Bugzilla bug list link and verify Fixed bugs. Read how the bug was produced and try to recreate it to verify the error doesn't happen anymore. 
# If the error is no longer occurring, select the "Mark bug as Verified" radio button and click the Commit button.  If the  error still occurs, select Reopen Bug and enter comments as to why the bug is being reopened (i.e. what is still failing), then click the Commit button.
'''OR - Run Litmus Test Cases'''
buildIds for August 22 Test Day
--------------------------------------------
                      Lightning/  Lightning/
OS      Sunbird    Tbird 1.5/2  Tbird 3.0
Linux    2006082206  2006082206  2006082206
Mac      2006082206  2006082009  2006082108
Windows  2006082207  2006082206  2006082206
# Go to [http://litmus.mozilla.org Litmus].
##Select Run Tests, the Product is Calendar. Your buildId should match the one in the table above. Verify that it does. Sunbird's buildId is in the "About Sunbird" window. Lightning's buildId is in its description in the Add-ons Manager.
##Choose a Test group and subgroup to test on (choose one that has a low percentage of Community Coverage).
##Follow the test steps and verify the function under test works as described in Expected Results.
## If the test case performs as expected, mark it as a "Pass".  If not, mark it as a "Fail" and then enter a bug (see below).
## When you've finished one test, go to the next test by clicking the test case name.
## Once you've completed all the tests in a group, click the "Submit All Results" button at the bottom.
'''OR - Perform Usability Testing'''
# Use Sunbird and Lightning like you normally would. Try to do everything an end-user might do, ranging from simple calendar operations to power user features. Try to think of actions that are not typical since you will most likely find a bug there.
'''BUG FOUND!'''
# When you think you've found a bug, try to reproduce it. Restart Sunbird/Lightning and do the same steps again. The most valuable bugs are those that can be reproduced.
# Once the bug has been reproduced, search for it or something similar on Bugzilla. This can easily be done by searching for the error message, window name, or some over identifying feature of the bug. If you found an existing bug that is just like the bug you found, read the comments. Maybe your situation is slightly different and your input could help developers find the origin of the bug.
# If you can't find the bug in Bugzilla, talk about it in the channel. Maybe others can confirm your bug or find a similar bug in bugzilla. This also gives moderators a chance to guide you to either provide more details, update an existing bug or file a new bug. 
# If you are certain you found a unique bug and have read the [https://bugzilla.mozilla.org/page.cgi?id=bug-writing.html Bug Writing Guidelines] and [https://bugzilla.mozilla.org/page.cgi?id=etiquette.html Bugzilla Etiquette], go ahead and file the bug in Bugzilla. Remember to be as specific as possible.
Some other ways you can help:
* Think of some test cases that aren't already included in Litmus (we love suggestions!).
* Ask your friends to join you. The more people help the better!
== Where do I find Litmus and Bugzilla? ==
* Litmus can be found at http://litmus.mozilla.org/
* A list of Calendar specific test cases can be found via the [http://litmus.mozilla.org/search_results.cgi?product=Calendar&locale=en-US search function]
* Bugzilla can be found at http://bugzilla.mozilla.org/
* A list of Calendar specific bugs (without enhancements) can be found [https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&short_desc_type=allwordssubstr&short_desc=&product=Calendar&long_desc_type=substring&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&keywords_type=allwords&keywords=&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=VERIFIED&resolution=---&bug_severity=blocker&bug_severity=critical&bug_severity=major&bug_severity=normal&bug_severity=minor&bug_severity=trivial&emailassigned_to1=1&emailtype1=exact&email1=&emailassigned_to2=1&emailreporter2=1&emailqa_contact2=1&emailtype2=exact&email2=&bugidtype=include&bug_id=&votes=&chfieldfrom=&chfieldto=Now&chfieldvalue=&cmdtype=doit&order=Bug+Number&field0-0-0=noop&type0-0-0=noop&value0-0-0= here]
* A list of already filed enhancement requests can be found [https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&short_desc_type=allwordssubstr&short_desc=&product=Calendar&long_desc_type=substring&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&keywords_type=allwords&keywords=&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&bug_status=VERIFIED&resolution=WONTFIX&resolution=---&bug_severity=enhancement&emailassigned_to1=1&emailtype1=exact&email1=&emailassigned_to2=1&emailreporter2=1&emailqa_contact2=1&emailtype2=exact&email2=&bugidtype=include&bug_id=&votes=&chfieldfrom=&chfieldto=Now&chfieldvalue=&cmdtype=doit&order=Bug+Number&field0-0-0=noop&type0-0-0=noop&value0-0-0= here]
Tips:
* Make sure you enter the correct information for your platform, version of Thunderbird, Lightning etc. Please include the full build identifier (i.e Thunderbird 1.5.0.5 (20060719) with Lightning 0.1+ (2006080907)).
* Be sure to read the [[Calendar:QA_TestDays#Testing_Instructions|Testing Instructions]] above before filing a bug.
* Feel free to ask on [[Calendar:QA_TestDays#Where_do_I_start.3F|IRC]] if you need help or have any more questions.
== How do I send feedback? ==
Please use email us at [mailto:dev-apps-calendar@lists.mozilla.org dev-apps-calendar at lists.mozilla.org] to send feedback about how the testday went, any overall thoughts about Sunbird and Lightning, and to follow up on issues found and bugs filed.  We really appreciate your help today and your feedback is very valuable.
== Rewards! ==
There will be two prizes given. Both prizes will be $25 gift certificates to the Mozilla Store. There are two categories for winners.
*Prize #1 goes to the person that completes the most tests. This means that they verify the most Bugzilla fixes and/or complete the most Litmus test cases.
*Prize #2 goes to the person that finds the best set of bugs. These bugs will be high-profile ones as deemed by the calendar QA team and the 0.3 release drivers as high profile, possibly blocking bugs. 
*As always, know that we appreciate everyone's contribution and you all have our heart felt gratitude. Thank you.

Revision as of 21:50, 28 September 2006

<< Back to Calendar QA Home Page

Next TestDay

Previous TestDays

Retrieved from "https://wiki.allizom.org/index.php?title=Calendar:QA_TestDays&oldid=34976"