Thunderbird:Testing: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(clean up nightly builds info)
(formatting)
 
(39 intermediate revisions by 8 users not shown)
Line 1: Line 1:
<small>[[Thunderbird:Home|<< Back to Thunderbird Home Page]]</small>
<small>[[Thunderbird:Home|<< Back to Thunderbird Home Page]]</small>
__NOTOC__
QA is a big space and there are many ways to help. Whether you have no experience or lots of experience, '''you''' can help make Thunderbird better. Your participation is needed and welcomed. Join in, and you too can feel you contributed to the  new release of Thunderbird. 


You are encouraged to have a look at the items below, but if you want a quick chat about it, please visit [[Thunderbird:Bugdays#Where|IRC #tb-qa]]. You might also be interested in the ongoing [http://perso.hirlimann.net/cgi-bin/mt/mt-search.cgi?blog_id=1&tag=interview&limit=20 QA Team Interviews].
You can help improve Thunderbird and other users - no prior experience is necessary.
Below are just a few ways you can contribute to the Quality of Thunderbird.  
__FORCETOC__
== '''Easy:''' ==


If you need other types of information about Thunderbird such as development, releases, add-ons, building, plans, etc, please visit [[Thunderbird|Thunderbird wiki]] and [http://www.mozillamessaging.com/ Mozilla Messaging].
These activities are simple, takes the least amount of time, and doesn't require any special skill.  


= How You Can Help in Thunderbird QA =
=== Beta testing ===


Ways you can help maintain and improve Thunderbird's quality:
Before Thunderbird is released to the general public, it goes through beta testing. To participate in this effort you'll need to run a [https://www.thunderbird.net/download/beta/ Beta version] which is different from the standard release version of Thunderbird.
* participate in community testing: [[Thunderbird:Testing#Bug_Days|Bug Days]], [[Thunderbird:Testing#Test Days|Test Days]], [[Thunderbird:Testing#QA New Releases|QA New Release Candidates]] of upcoming versions, i.e. point releases, Thunderbird 3 (3.0, 3.1 aka 3.next, etc), betas, and/or alphas.
The [https://www.thunderbird.net/download/beta/ Beta web page] has advice and links to many helpful resources, including a beta topicbox group for advice and assistance.
* run [[Thunderbird:Testing#Daily_Testing|nightly builds and early releases]] on a daily basis
** Nightly testers tend to hang out in [http://forums.mozillazine.org/viewforum.php?f=29 Thunderbird Builds forum]
** How to: [[Thunderbird:Testing:Get_A_Debug_Thunderbird_Hang_Stack|get a stack off a debug Thunderbird hang]]
* stay informed of QA activities and announcements
** Talk to and get help from other testers in IRC channel #tb-qa. [[Thunderbird:Bugdays#Where|IRC Instructions]]
** subscribe to [https://mail.mozilla.org/listinfo/thunderbird-testers thunderbird-testers announcement list]
** mozilla.dev.apps.thunderbird [news://news.mozilla.org/mozilla.dev.apps.thunderbird newsgroup], [http://groups.google.com/group/mozilla.dev.apps.thunderbird web], [https://lists.mozilla.org/listinfo/dev-apps-thunderbird mailing list]
** mozilla.dev.quality [news://news.mozilla.org/mozilla.dev.quality newsgroup], [http://groups.google.com/group/mozilla.dev.quality web], [https://lists.mozilla.org/listinfo/dev-quality mailing list]
** Planet: [http://planet.mozillamessaging.com/ subscribe to Mozillamessaging's Planet], [http://planet.mozilla.org moco planet]
** [http://quality.mozilla.org/blog QMO]
* watch in real time:
** bug activity via IRC [http://www.mibbit.com/chat/?server=irc.mozilla.org&channel=%23tb-bugs #tb-bugs] (pick a nickname, then click Go)
** [http://wiki.mozilla.org/QA/Community/Bug_Watchers bug activity via bugmail] (and [[Thunderbird:Bug_Triage|help triage them]])
** [http://hg.mozilla.org/comm-central/ comm-central] patch/fix [http://hg.mozilla.org/comm-central/atom-log checkins via RSS]
* Participate in [[Thunderbird:Testing#Patch Love|Patch Love]]
* help with Tests, QA support and Documentation
** develop [https://developer.mozilla.org/en/Thunderbird/Thunderbird_MozMill_Testing MozMill Tests]
** improve [[Thunderbird:Testing#Litmus Tests|Litmus Tests]]
** improve [[Thunderbird:Testing#Documentation and Procedures|Documentation and Procedures]]
** help integrate Thunderbird QA into the [[Thunderbird:Testing#QMO|QMO initiative]]
** offer to lead or co-lead any of the QA areas, and suggest new QA initiatives
* [http://www.mozilla.org/support/thunderbird/bugs Report Bugs]
* Crashes:
** If you crash, please fill in "Add a comment" before clicking Crash Reporter's "Restart".
** Don't like crashes? Help [http://crash-stats.mozilla.com/query/query?product=Thunderbird&version=Thunderbird%3A3.0b3&date=&range_value=1&range_unit=weeks&query_search=signature&query_type=exact&query=&do_query=1 monitor], report, [https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&product=MailNews+Core&product=Thunderbird&resolution=---&bug_severity=blocker&bug_severity=critical&bug_severity=major&bug_severity=normal&chfieldto=Now&field0-0-0=short_desc&type0-0-0=substring&value0-0-0=crash&field0-0-1=keywords&type0-0-1=substring&value0-0-1=crash track and recreate/resolve crashes]
* contribute suggestions, participate in planning and development by:
** [[Thunderbird:Testing#Documentation and Procedures|editing any wiki page]] that is unclear or that can be improved
** posting in mozilla.dev.quality [news://news.mozilla.org/mozilla.dev.quality newsgroup ], [http://groups.google.com/group/mozilla.dev.quality web], [https://lists.mozilla.org/listinfo/dev-quality mailing list]
** Discussion of development efforts in [news://news.mozilla.org/mozilla.dev.apps.thunderbird mozilla.dev.apps.thunderbird dev newsgroup]


General Information:
=== Finding regressions ===
* [http://www.gerv.net/hacking/how-to-ask-good-questions/ How to ask good questions]
While testing or searching bugzilla you might find some portions of Thunderbird that previously worked but do not work anymore. Bug reports will have the '''regression''' keyword.  
* [https://developer.mozilla.org/en/Bug_writing_guidelines Bug writing guidelines]
* [[Thunderbird:Bug_Triage|Bug triage]]
* [[Thunderbird:Bug_Queries|Bug queries]]
* [http://www.mozillamessaging.com/en-US/about/get-involved/ Getting Involved] in Mozilla Messaging
* [[Thunderbird|Thunderbird 3]]


Triage references:
For these bugs it is very helpful to know the "regression range", the 1-day range of changes that broke Thunderbird.  A bug whose regression range has not been determined also has a bugzilla keyword '''regressionrange-wanted'''.
* [[Thunderbird:Testing:Memory_Usage_Problems|Performance Problems, Memory Usage]]
* [[Thunderbird:Testing:Shutdown_Hang]]
* [[Thunderbird:Debugging_Gloda]]


= Bug Days =
You can find the regression range using a wonderful semiautomated tool called [https://mozilla.github.io/mozregression/ mozregression], and hopefully identify the code which caused the regression behavior.


* Help triage, improve and reduce the bugs reported to our bugzilla database. [[Thunderbird:Bugdays|Bugday information]].  [[Thunderbird:QA_Days|Previous / planned QA days]].
== '''Moderate:''' ==
* Suggest bug groupings to be triaged in [[Thunderbird:Bugdays_Planning]]


= Test Days =
=== Performance Profiles ===


Test new features, specific features, or new releases [[Thunderbird:Testdays|Testdays information]]. [[Thunderbird:QA_Days|Previous / planned QA days]].
Bugs which are performance related, with the "perf" keyword, can be profiled using a tool which traces the code being used.  [https://support.mozilla.org/en-US/kb/profiling-thunderbird-performance This page] has details on running the tool.  More information to come.


= Patch Love =
=== Maintaining Thunderbird's bug reports ===
[https://bugzilla.mozilla.org/ Bugzilla] is the database / tool that tracks reported defects. Working in this area requires some experience using Thunderbird, and the ability to determine what the user reporting a problem might be seeing. You don't need great technical skill. It just takes a little time. More information on this activity can be found [[Thunderbird:Testing/bugzilla|here]].


[https://wiki.mozilla.org/Thunderbird:PatchLove Give love] so that patches get checked in, and patch authors stay involved in Thunderbird.
== '''Difficult:''' ==


= QA Wanted  =
=== Daily build testing ===
Some bugs need more work to be usable for our developers. Those bugs are marked with the Keyword [https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&short_desc_type=allwordssubstr&short_desc=&classification=Client+Software&classification=Components&product=MailNews+Core&product=Thunderbird&long_desc_type=substring&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=qawanted&keywords_type=allwords&keywords=&resolution=DUPLICATE&resolution=---&emailassigned_to1=1&emailtype1=exact&email1=&emailassigned_to2=1&emailreporter2=1&emailqa_contact2=1&emailtype2=exact&email2=&bugidtype=include&bug_id=&votes=&chfieldfrom=&chfieldto=Now&chfieldvalue=&cmdtype=doit&order=Reuse+same+sort+as+last+time&field0-0-0=noop&type0-0-0=noop&value0-0-0= qawanted]. The idea there is to figure out a way to reproduce, find a proper test case and then add that information to the bug.


= QA New Releases =
This requires running [https://www.mozilla.org/en-US/thunderbird/channel/ Daily version] which is different from the standard release version of Thunderbird.  Daily is a difficult environment, because you get a new version almost every day, with updated code.  Sometimes the daily build is broken by bugs. Otherwise, the challenges are similar to running a beta version. The [https://thunderbird.topicbox.com/groups/daily daily topicbox group] is available for advice and assistance.


Release candidates for new versions/releases are run through QA (Quality Assurance) testing to ensure we ship high quality code. To properly QA a new release we need many volunteer like you from each OS. To be notified of testing opportunities, please sign up for the [https://mail.mozilla.org/listinfo/thunderbird-testers testers mailing list] and/or you can set up to watch wiki pages at [[Releases|Upcoming / In-progress New Releases]] | [http://www.mozilla.com/en-US/thunderbird/releases/ Past releases].
<!-- == '''Challenging:''' Writing automated tests ==


[http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/3.0rc1-candidates/build1/ 3.0 RC1 builds] are available for testing. Please file any new bugs found during the testing period as blocking on {{bug|524434}}.
This is the most technical item. It requires a good amount of technical knowledge. But if you do not have the knowledge this is a good way to learn a lot of technical stuff. This involves understanding programming or learning about it.


= Daily Testing =
More information on writing tests is available here.-->__NOTOC__
== Ask questions on Matrix ==
If you want to know more, ask in the Thunderbird support topic ''#thunderbird:mozilla.org'' on [https://wiki.mozilla.org/Matrix Matrix]. Bear in mind volunteers may not respond immediately, so please be patient.


If you can handle some risk**, a fantastic way to help speed development and improve the Thunderbird's quality is to continuously run an [http://www.mozillamessaging.com/en-US/thunderbird/early_releases/ early release] or [ftp://ftp.mozilla.org/pub/thunderbird/nightly/latest-comm-1.9.2/ nightly build] and submit bugs for problems that you find. And [http://forums.mozillazine.org/viewforum.php?f=29 talking to other nightly users is easy].
<!-- Backup. I don't think that we need it:
 
<sup>[[#number1|1]]</sup>
'''** Ability to "handle the risk" means you have great backups of your profiles and mail store, and have some tolerance for risk.''' Dataloss is rare, but is a possibility. Stability is amazingly great, but not guaranteed. And  fixes to bugs in these builds don't come overnight.  But many people run unsupported builds. Nightlies and some early releases are code named Shredder. Shredder is an unbranded state of Thunderbird code base. Development code eventually ships as a future version, like 3.2.
#<div id="number1"> this is the #tb−qa channel on </div>
 
Some extensions won't work. But many like enigmail and [http://www.mozilla.org/projects/calendar/lightning/download.html calendar] work if you use the right download for trunk or nightly. Many work if you override the version check and carefully test. Determine one at a time which work using [https://addons.mozilla.org/en-US/thunderbird/addon/6543 Nightly Tester Tools extension].  You can also [http://kb.mozillazine.org/Run_multiple_copies_of_Thunderbird_at_the_same_time run the current "production" version of Thunderbird at the same time].
 
See what fixes have checked in at [http://hg.mozilla.org/comm-central/ comm-central] and watch the changes via [http://hg.mozilla.org/comm-central/atom-log RSS].  [http://forums.mozillazine.org/viewforum.php?f=29 Build forum] is also a good source of information.
 
= Gristmill/MozMill Tests =
 
[[QA/TDAI/Gristmill|Gristmill/MozMill]] is a tool to use for UI automation in any mozilla platform developed application. It can provide Thunderbird with the capability to quickly repeat tests in a consistent manner, being another aid to avoid faults creeping into the application with the constant development.
 
Gristmill is still being developed, however the basic concepts are there and it is possible to run tests manually, see [http://ccgi.standard8.plus.com/blog/archives/36 this link] for an example within Thunderbird.
 
The Gristmill scripts are written in Javascript, there is a [[QA/TDAI/Gristmill/Mozmill Tutorial|tutorial]] available and it should be noted that you do not need an in-depth knowledge of Thunderbird's internals.
 
Gristmill is still new, and we'd be interested in getting some basic scripts together and setting up some kind of testing architecture. If you're interested, please contact [[User:Standard8]] direct.
 
We are currently putting together a more detailed, [[MailNews:Automation_With_Gristmill|MailNews specific page for Gristmill]].
 
= Litmus Tests =
 
Litmus is used to organize and assist human (i.e. non-automated) testing of mozilla software products. We need your help to:
* add new tests - pick from the list of [http://tinyurl.com/54s52j bugs with in-litmus? flag]
* fix [http://bit.ly/ccRnBp unvetted v3x tests reported to be "unclear", sorted by summary/title]
* run tests (you can run these tests at any time)
** [https://litmus.mozilla.org/run_tests.cgi?test_run_id=30 1.9 Branch] - The baseline test plan for Thunderbird 3.0 builds.
** [https://litmus.mozilla.org/run_tests.cgi?test_run_id=5 1.8 Branch] - The baseline test plan for Thunderbird 2.0 builds.
 
** [https://litmus.mozilla.org/run_tests.cgi?test_run_id=6 Trunk] - The baseline test plan for Thunderbird accessibility and localization L10n.
 
** Work in [[Thunderbird:Testing/wip|progress tests]]
 
If you need more information, please post on IRC or in newsgroup.
<!-- === Release Testing Matrix ===
 
{| class="fullwidth-table"
!
! Windows
! Linux
! Mac
|-
| TBD
|
|
|
|}
-->
-->


= Documentation and Procedures =
<small>Backup (for service): [[User:Tsk/bckup|Old Content]]</small>
 
[[category:Thunderbird]]
Most of our documentation is on Mozilla Wiki in a [[Thunderbird:Home|few]] [[Thunderbird:Help_Documentation:Base|locations]]. Anyone can edit and improve these pages, you just need an easy to obtain [[Special:Userlogin|wiki account]].
 
With an account, you can also [http://wiki.mozilla.org/index.php?title=Thunderbird:Testing&action=watch watch] for updates to pages that might interest you. Some wiki pages, like [[Thunderbird:QA_Days|QA days]], change for every new date ... so watch is a great notification mechanism.
 
= QMO =
 
QMO, [http://quality.mozilla.org quality.mozilla.org], is a new facility providing integrated QA environment. QMO are preparing to roll out this functionality for Firefox, but for Thunderbird to leverage the QMO infrastructure, we need volunteers to step forward to review, plan and build the Thunderbird specific pieces.
 
= Test Automation =
 
Ideas for Thunderbird's unit test framework can be [[Thunderbird:Unit_Test_Ideas|found here]].
 
[[MailNews:Home_Page#Automatic_Testing|Automated testing for Mailnews]]
 
 
[[category:Thunderbird|*]]

Latest revision as of 16:25, 15 January 2024

<< Back to Thunderbird Home Page

You can help improve Thunderbird and other users - no prior experience is necessary. Below are just a few ways you can contribute to the Quality of Thunderbird.

Easy:

These activities are simple, takes the least amount of time, and doesn't require any special skill.

Beta testing

Before Thunderbird is released to the general public, it goes through beta testing. To participate in this effort you'll need to run a Beta version which is different from the standard release version of Thunderbird. The Beta web page has advice and links to many helpful resources, including a beta topicbox group for advice and assistance.

Finding regressions

While testing or searching bugzilla you might find some portions of Thunderbird that previously worked but do not work anymore. Bug reports will have the regression keyword.

For these bugs it is very helpful to know the "regression range", the 1-day range of changes that broke Thunderbird. A bug whose regression range has not been determined also has a bugzilla keyword regressionrange-wanted.

You can find the regression range using a wonderful semiautomated tool called mozregression, and hopefully identify the code which caused the regression behavior.

Moderate:

Performance Profiles

Bugs which are performance related, with the "perf" keyword, can be profiled using a tool which traces the code being used. This page has details on running the tool. More information to come.

Maintaining Thunderbird's bug reports

Bugzilla is the database / tool that tracks reported defects. Working in this area requires some experience using Thunderbird, and the ability to determine what the user reporting a problem might be seeing. You don't need great technical skill. It just takes a little time. More information on this activity can be found here.

Difficult:

Daily build testing

This requires running Daily version which is different from the standard release version of Thunderbird. Daily is a difficult environment, because you get a new version almost every day, with updated code. Sometimes the daily build is broken by bugs. Otherwise, the challenges are similar to running a beta version. The daily topicbox group is available for advice and assistance.


Ask questions on Matrix

If you want to know more, ask in the Thunderbird support topic #thunderbird:mozilla.org on Matrix. Bear in mind volunteers may not respond immediately, so please be patient.


Backup (for service): Old Content

Retrieved from "https://wiki.allizom.org/index.php?title=Thunderbird:Testing&oldid=1249439"