QA

What We Use

We typically use nightly optimized (non-debug) builds for daily usage. However, we also use the release builds (of course!), as well as older builds when trying to narrow down regression windows.

• For nightly builds, check out any of the mirrors, then drill down to the <product_name>/nightly/ directory. While you can go to <product_name>/latest-* directories, the problem there is that you don't necessarily known when those builds were made. It's best to access the specific build-date directory (e.g., 2005-03-17-08-trunk), to know what you're grabbing.
• For older builds not listed in the mirror pages, check out the archives at http://archive.mozilla.org/pub/
• For release builds, simply go to any of the mirrors and drill down to <product_name>/releases/ and select the appropriate directories for version, platform and locale.

Bugzilla

We depend on Bugzilla for filing and tracking bugs and features. We frequently use the query tools, both the "Advanced Search" and "Find a Specific Bug" queries. With the bug count reaching 300,000, there are a couple ways to see what's been frequently reported and duplicated:

• Most frequently reported bugs query
• Most recently duplicated bug query

Testrunner / Litmus

We currently use Testrunner at http://testrunner.mozilla.org for test development and execution of various types of test runs like smoketests and basic functional tests (BFT's). To view the following test plans you need a Testrunner login.

• Firefox smoketests
• Thunderbird smoketests
• Firefox BFT's
• Thunderbird BFT's

We are also currently in the process of developing Litmus as a replacement for Testrunner. Litmus will build upon the testcase management aspects of Testrunner, adding the searchability, reporting, and trending tools that are critical to our QA work. More information about Litmus development is available in the Litmus wiki.

Talkback

When an application crashes, we use Talkback to examine the crash information. A publically available Talkback server can be accessed at http://talkback-public.mozilla.org

There are a number of tools available there:

• Reports (http://talkback-public.mozilla.org/reports) - Browse topcrash data for all Mozilla products. Topcrash lists, stack traces, user comments and urls are available to help users reproduce crash bugs.
• FastFind - Lookup individual crash incidents.
• QuickSearch - Run customized queries to get a better understanding of specific crashes.

How To:

• Find your Talkback incident id
• Pick a component for crash bugs (http://www.mozilla.org/quality/help/pick-a-component-for-crashers.html)
• Talkback Talkback Analysis, or how to log a topcrash bug
• Use FastFind results
• Use QuickSearch results
• see also Talkback article at kb.mozillazine.org.

ToDo List:

• Clean up and update Talkback reporting system and website
• Create Talkback/release database to track builds
• Bug fixes and enhancements for Talkback FastFind/QuickSearch tools.

Development tools

We also use several development tools for tracking changes, especially useful for narrowing down regression windows!

• Tinderbox (http://tinderbox.mozilla.org/showbuilds.cgi) to visually display our continuous build system. Great to see who checked in what, the state of the build, as well as quick links to automated performance tests.
• LXR (http://lxr.mozilla.org) to examine the source code.
• Bonsai (http://bonsai.mozilla.org) to narrow down when changes were made.

Automation

In an effort to the improve the testing potential of our team, we are constantly investigating ways to leverage automated testing tools to offload some of the regression testing burden from our core testers.

• A listing of the various candidate software we have considered and the status of each.