MozillaWiki

QA/Firefox3/TestResults/M9

< QA‎ | Firefox3‎ | TestResults
Revision as of 21:46, 13 November 2007 by Deliriousgreen (talk | contribs) (fix bug numbers (6+7 transposed))

Back to QA Fx3 Home

Firefox 3 M9 Test Results

Nightly Smoke Test Results on 11/6/2007

Test Results from the nightly smoke test before the official M9 builds.

Contact Info

Build: John O'Duinn (joduinn)
QA: Tony (tchung)

Schedule

Code Freeze
11/2
Code Build
11/7
QA test signoff
ETA 11/13

Release Notes

  • TBD

Build Location and IDs

RC1:

  • Mac Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9b1) Gecko/2007110618 Firefox/3.0b1
  • Vista Build ID: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b1) Gecko/2007110703 Firefox/3.0b1
  • XP Build ID: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b1) Gecko/2007110703 Firefox/3.0b1
  • Linux Build ID: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9b1) Gecko/2007110618 Firefox/3.0b1


RC2:

  • Mac Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9b1)Gecko/2007110810 Firefox/3.0b1
  • Vista Build ID: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b1)Gecko/2007110811 Firefox/3.0b1
  • XP Build ID: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b1)Gecko/2007110811 Firefox/3.0b1
  • Linux Build ID: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9b1) Gecko/2007110810 Firefox/3.0b1

RC3:

  • Mac Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9b1) Gecko/2007110903 Firefox/3.0b1
  • Vista Build ID: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b1) Gecko/2007110904 Firefox/3.0b1
  • Linux Build Id: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9b1) Gecko/2007110903 Firefox/3.0b1
  • XP Build ID: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b1)

Gecko/2007110904 Firefox/3.0b1

Features landed

Firefox

  • List coming soon

Gecko & Platform

  • List coming soon

Strategy

  • Areas Covered:
    • Smoketests
    • FFTs
    • M9 and past focused Tests
    • Top Extensions
    • Top Plugins
    • Top Sites (manual by QAE, automation by bc)
    • Memory Leak Testing (handled by Tomcat on debug builds)
    • Lite L10N spotchecks only. (need list of supported locales)
    • M9 bug verifications
  • Areas Not Covered:
    • L10N
    • Distribution
    • Updates

Tests are ran against en-US locale only.

Test Results

Types of Tests Win XP Win Vista (include Vista Litmus Test Suite) Mac Tiger Mac Leopard Linux
Smoketest Tomcat - PASS Tomcat - PASS Tracy - PASS N/A Stephend - PASS
FFT Tomcat - in progress Tomcat (partial) - PASS Tchung - RESULT Marcia - RESULT Tomcat - PASS
Top Sites Testing (see below) Juanb - in-progress
bc (automation) - results 		N/A Juanb - in-progress N/A bc (automation) - results
Top Extensions Testing (see below) Abillings - RESULT N/A Abillings - RESULT N/A Tracy - RESULT
Top Plugins Testing (see below) Stephend - PASS N/A Stephend (Marcia, really, in her FFT testing; thanks Marcia!) - PASS N/A Tracy - RESULT
l10N Pri 1 locales Tracy (en-GB, pl, fr) - RESULT,
Tomcat (de) - RESULT 		N/A Tchung (ja, es-ES) - PASS N/A N/A
Bug Verifications Everyone Everyone Everyone Everyone Everyone
  • Windows XP Smoketest, en-US, - Tomcat
    • Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b1) Gecko/2007110703 Firefox/3.0b1
    • PASS - addons testing currently not possible, because of no addons ready for maxversion b1

Top Sites Testing

Select the top 50 sites, and run through a combination of various web applications.

Testcases

  • Visit various sites on the list, verifying web compatibility
  • Detect Layout issues, Font issues, formatting problems, broken functionality
    • Some baseline comparisons is to use Fx 2.0.0.9, Safari, and Trunk nightly builds
  • Change browser font sizes, languages and verify web compatibility
  • Operate end to end functionality of a website. Examples:
    • track an auction item on ebay
    • save a book on amazon checkout
    • register for a new email account on Yahoo Mail
    • Create a document on Google docs
    • Visit a secure site and perform online banking

Top Extensions Testing

Select a few top extensions from this list and run tests to install/update/uninstall/disable extensions. Available for download at AMO.

Or, refer to this list: Most-Used Addons

Testcases:

  • Enabling extensions and confirming web compatibility
  • Disabling or Uninstalling extensions and confirming proper messaging and behavior
  • Load about 10+ extensions, and fire them up all at once, verifying functionality and performance
  • Updating old extension versions to the latest if applicable
  • Switch profiles with different extensions, and confirm they do not get carried over.

Top Plugins Testing

Select a few top plugins on the web. Use various websites supporting these plugins. Available for download at AMO.

Consider:

  • Adobe PDF
  • Flash
  • Java
  • Shockwave
  • Embedded Win Player
  • Embedded Divx player
  • Quicktime player
  • Viewpoint on both Vista and XP
  • Ipix (360 degree view)

Testcases:

  • Enabling Plugins and confirming web compatibility
  • Disabling Plugins and confirming proper messaging and behavior
  • Check about:plugins and Addon Managers to verify plugins still exist
  • Updating old plugin versions to the latest

Issue found:

Pre-existing bug 328675

L10N P1 Spotchecks

Visual inspection: Go through the P1 list of locales, and run a spotcheck to see if everything visually looks fine. Refer to L10N Requirements for more specifics.

Automation: We'll us beta 1 to create the reference files. Will be running minotaur in Beta 2 and beyond.

P1 Locales to be tested:

  • English (en-GB)
  • French (fr)
  • German (de)
  • Japanese (ja)
  • Polish (pl)
  • Spanish (es-ES)

Testcases and Results are tracked here.

Bug Verifications

Bugs that are marked RESOLVED FIXED that are checked in for beta. Please verify them.

Detailed Results

Tchung Results
marcia's leopard bugs and problem areas
Tomcat Results
  • Vista FFT - PASS
    • Bugs filed:
      • bug 403497 Places Toolbar Folder not renamed when language version is changed
      • bug 403401 Wrong Link to Store "Firefox on CD" on www.mozilla.org
      • bug 403257 Localized Extension description not maintained when the Add-on is disabled
      • bug 403246 Last line of Anti Phishing Warning Page contains de and en-US text
      • bug 403185 "Open All in Tabs" appends, rather than replaces, current tabset
      • bug 403628 Failed to load module "gnomebreakpad - Error output on Trunk Start with Fedora 8 - the problem on this bug is, that also the Breakpad System itself is broken on Fedora 8 - see https://bugzilla.mozilla.org/show_bug.cgi?id=392919 - this should block a beta 2 i guess


Stephend Results
  • Prexisting bug 328675 - I nominated it for "blocking-1.9?", but jst claims plugins aren't really being actively worked on :-(
Juanb results
(topsite)
11/9/07 Testday Results

Full Summary here. Highlights:

  • 1300 litmus tests by Community
  • 500 litmus tests by l10n Community
  • 56 failed testcases
  • 60 unclear/broken testcases
  • 50 new Community Members total, and 35 using Litmus

Live Spot Checks for M9

When we go live at 3 PM it would be great team members could download an M9 build quickly and make sure everything looks ok.

  • Windows XP -> Tomcat - TBD
  • Windows Vista -> abillings TBD
  • Mac PPC/Intel-> tchung TBD
  • Mac Leopard -> Marcia
  • Linux -> stephend TBD

Blocker Bugs for M9

Refer to dev-tracking bugs at: [M9 firefox]|[M9 platform]

Retrieved from "https://wiki.allizom.org/index.php?title=QA/Firefox3/TestResults/M9&oldid=74300"