QA/Firefox3.5/TestPlan/Beta2

From MozillaWiki
< QA‎ | Firefox3.5‎ | TestPlan
Jump to: navigation, search

« QA/Firefox3.1/TestPlan

Firefox 3.1 Beta2 Test Results

The covered testing for this release will focus on smoketests, ffts, l10n, top extensions, top plugins, updates, and spotchecks.

Related Pages

Firefox 3.1 Tracking Site

Contact Info

PM: Mike Beltzner (beltzner)
Build: John O'Duinn (joduinn)
QA: Tony / others

Schedule

Code Freeze
11/4
String Freeze
11/10
Builds hand to QA
11/25
QA test signoff
12/5
  • See Test Coverage below

Build Location and IDs

Download:

Builds:

  • Windows XP: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b2) Gecko/20081201 Firefox/3.1b2
  • Mac: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b2) Gecko/20081201 Firefox/3.1b2
  • Linux: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b2) Gecko/20081201061202 Firefox/3.1b2

Test Coverage

  • Areas Covered:
    • Smoketests
    • BFTs
    • L10n
    • Updates
    • Top Extensions
    • Accessibility Spot check
    • Spotchecks & Webpages
    • Bug Fixes
  • Areas Not Covered:
      • FFTs
      • Topsites (covered in nightly automation)
      • Top Themes
      • Top Plugins

Test Results

Types of Tests Windows XP Mac OSX Linux
Smoketests marcia - PASS Tracy - PASS <thead> </thead> <tbody> </tbody>
Joel - PASS
BFTs
marcia/Tony - PASS
tracy - PASS
Joel - PASS
-
L10n
Tony - PASS
Tony - PASS
Tony - PASS
-
Top Extensions
Murali - PASS
Murali - PASS
Murali - MIXED
-
Accessibility
Marco - PASS
N/A
Marco - PASS
-
Updates
Murali
Betatest - PASS
releasetest - PASS
Beta - PASS
Tony
Betatest - PASS
releasetest - PASS
Beta - PASS
Tracy/Murali
Betatest - PASS
releasetest - PASS
Beta - PASS
-
Live Spotchecks & Webpages
Tony - PASS
Tony - PASS
Murali - PASS
}

Smoketest Testing

Lite verification tests (or smoketests) will be first ran when the builds are ready. Tests are available on litmus under the Firefox 3.1 test suite. In any case the Smoketests result, we will raise blockers asap before proceeding with the bfts.

Please record your detailed results in the section below.

BFT / FFT Testing

BFTs are a more comprehensive set of litmus tests that cover the build testing after smoketests result. Tests are available on litmus under the Firefox 3.1 test suite.

FFTs are a full comprehensive set of litmus tests that extend coverage in areas that BFTs dont often get to. These are run at least once during the major milestone timeline.

If you are running a BFT or FFT, Please record your results in the Detailed Results section

Updates Testing

We will be doing updates of all Alpha users to Beta2. We will be checking complete updates, and resultover scenarios.

Will be available on betatest, releasetest, and beta channels.

Testcases and Results are tracked here.

L10N

Visual inspection: Go through the P1 list of locales, and run a spotcheck to see if everything visually looks fine. Refer to L10N Requirements for more specifics.

Testcases and Results are tracked here.

Top Extensions Testing

Select a few top extensions from AMO and perform the following testcases. See Fx3 Compatibility Addons List Here.

Also, this is the top 50 most popular addons voted by mozdev.

Testcases:

  • Enabling extensions and confirming web compatibility
  • Disabling or Uninstalling extensions and confirming proper messaging and behavior
  • Load about 10+ extensions, and fire them up all at once, verifying functionality and performance
  • Updating old extension versions to the latest if applicable
  • Switch profiles with different extensions, and confirm they do not get carried over.

Testcases and Results are tracked here.

Accessibility

Ensure UI pieces such as awesomebar, ctrl-tab, private browsing, etc.. are working correctly and follows a11y standards.

Live Spotchecks & Webpages

1) Verify the following webpages that gets pushed to staging and then to the live sites. Look for content messaging, spelling mistakes, broken links & imnages, etc..

  • Firstrun page
  • Whatsnew Page
  • Releasenotes page

2) Check live site download links on all platforms. Please cover en-US, and 2 or 3 additional Tier 1 locales as well. Pay attention to the following areas:

  • Check for signed installer and applications (windows only)
  • Firstrun page, whatsnew page, Home page, Search plugin
  • Install, uninstall
  • General spotchecks and browser functionality
  • Confirm Build IDs

Fixed Bugs Verifications

The following bugs are fixed for Beta 2 and need verifications:

Testday

A Beta2 testday will be held and call for the community support to assist with testing. Activities include executing litmus testcases, bug verifications, and focus on the new feature areas that are being implemented. More can be found on QMO.

Detailed Results

Smoketests Results

Windows XP [marcia]

  • PASS
  • Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b2) Gecko/20081125 Firefox/3.1b2
    • Tear away tab feature does not seem as smooth as when running on Mac. Win XP machine does not have a lot of memory so this may be the cause.

Mac [tracy]

  • 11/15 - Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.1b2) Gecko/20081125 Firefox/3.1b2 - PASS
    • Results in Litmus
    • Didn't run install ext or theme test cases. none compatible, not even nightly tester tools.

Linux [jmaher]

  • Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b2) Gecko/20081125 Firefox/3.1b2 - PASS

BFT Results

Windows XP [marcia/Tony]

  • Filed bug 467601 - long bookmark descriptions will hide tagging icon and text

Mac [tracy]

  • Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.1b2) Gecko/20081201 Firefox/3.1b2 - PASS
    • edge case about:config bug found in litmus [7415] failed: bug [467565] Private browsing doesn't respect "Quit Firefox" Dialog choice if browser.privatebrowsing.autostart is set to true.

Linux [jmaher]

  • Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b2) Gecko/20081125 Firefox/3.1b2 - PASS