QA/Firefox3.5/TestPlan/RC1: Difference between revisions

From MozillaWiki
< QA‎ | Firefox3.5‎ | TestPlan
Jump to navigation Jump to search
m (Created page with '<small>« QA/Firefox3.5/TestPlan</small> = Firefox 3.5 RC1 Test Results = The covered testing for this release will focus on smoketests, bfts, l...')
 
mNo edit summary
Line 8: Line 8:
[http://wiki.mozilla.org/Firefox3.5 Firefox 3.5 Tracking Site]
[http://wiki.mozilla.org/Firefox3.5 Firefox 3.5 Tracking Site]
* [https://wiki.mozilla.org/Releases/Firefox_3.5b4 Release Schedule]
* [https://wiki.mozilla.org/Releases/Firefox_3.5b4 Release Schedule]
* [https://wiki.mozilla.org/Firefox3.5/Features beta 4 features]
* [https://wiki.mozilla.org/Firefox3.5/Features RC1 features]


== Project contacts ==  
== Project contacts ==  
PM: Mike Beltzner (beltzner) <BR>
PM: Mike Beltzner (beltzner) <BR>
Build: Nick Thomas  (nthomas)<BR>
Build: Nick Thomas  (nthomas)<BR>
QA: Tony Chung (tchung)
QA: Tony Chung (tester)


== Schedule ==  
== Schedule ==  
Line 55: Line 55:
|-
|-
| Smoketests
| Smoketests
| Henrik - '''PASS''' (unsigned) <br> Timr - '''PASS''' (signed)
| tester - '''result''' (unsigned) <br> Timr - '''result''' (signed)
| Aakash/Timr - '''PASS'''
| Aakash/Timr - '''result'''
| Tracy - '''PASS'''  <small>Though randomly crashed a couple times.</small>
| tester - '''result'''  <small>Though randomly crashed a couple times.</small>
|-
|-
| BFTs
| BFTs
| Henrik/Murali - '''PASS'''  
| tester/Murali - '''result'''  
| Marcia/Tchung - '''PASS'''
| tester/tester - '''result'''
| Aakashd/Tracy - '''PASS'''  
| tester/tester - '''result'''  
|-
|-
| L10n
| L10n
| Tchung - '''PASS'''
| tester - '''result'''
| Tchung - '''PASS'''
| tester - '''result'''
| Tchung - '''PASS'''
| tester - '''result'''
|-
|-
| Updates
| Updates
| Tracy<BR> '''Betatest - PASS''' <BR> '''releasetest - PASS''' <BR> '''Beta - result'''  
| tester<BR> '''Betatest - result''' <BR> '''releasetest - result''' <BR> '''Beta - result'''  
| Henrik/Murali<BR> '''Betatest - PASS''' <BR> '''releasetest - PASS''' <BR> '''Beta - PASS'''  
| tester<BR> '''Betatest - result''' <BR> '''releasetest - result''' <BR> '''Beta - result'''  
| Aakashd/tester? <BR> '''Betatest - PASS''' <BR> '''releasetest - PASS''' <BR> '''Beta - PASS'''
| tester<BR> '''Betatest - result''' <BR> '''releasetest - result''' <BR> '''Beta - result'''
|-
|-
| Additional Focused Testing
| Additional Focused Testing
|  Fx-team, QA team - '''PASS'''
|  QA team - '''result'''
|  Fx-team, QA team - '''PASS'''
|  QA team - '''result'''
|  Fx-team, QA team - '''PASS'''
|  QA team - '''result'''
|-
|-
| Topsite Testing
| Topsite Testing
|  Tomcat - '''PASS'''
|  tester - '''result'''
|  Tomcat - '''PASS'''
|  tester - '''result'''
|  Tomcat - '''PASS'''
|  tester - '''result'''
|-
|-
| Staging/Live Webpages & Spotchecks & Webpages
| Staging/Live Webpages & Spotchecks & Webpages
| <web tester>, henrik - '''result'''
| tester - '''result'''
| <web tester>, marcia - '''result'''
| tester - '''result'''
| <web tester>, Aakashd - '''PASS'''
| tester - '''result'''
|}
|}


=== Smoketest Testing ===
=== Smoketest Testing ===
Lite verification tests (or smoketests) will be first ran when the builds are ready.  Tests are available on litmus under the [https://litmus.mozilla.org/run_tests.cgi?test_run_id=31 Firefox 3.1 test suite].  In any case the Smoketests result, we will raise blockers asap before proceeding with the bfts.
Lite verification tests (or smoketests) will be first ran when the builds are ready.  Tests are available on litmus under the [https://litmus.mozilla.org/run_tests.cgi?test_run_id=35 Firefox 3.5 test suite].  In any case the Smoketests result, we will raise blockers asap before proceeding with the bfts.


Please record your detailed results in the section [https://wiki.mozilla.org/QA/Firefox3.5/TestPlan/Beta4#Detailed_Results below].
Please record your detailed results in the section [https://wiki.mozilla.org/QA/Firefox3.5/TestPlan/RC1#Detailed_Results below].


=== BFT / FFT Testing ===
=== BFT / FFT Testing ===
Line 100: Line 100:
FFTs are a full comprehensive set of litmus tests that extend coverage in areas that BFTs dont often get to.  These are run at least once during the major milestone timeline.   
FFTs are a full comprehensive set of litmus tests that extend coverage in areas that BFTs dont often get to.  These are run at least once during the major milestone timeline.   


If you are running a BFT or FFT, Please record your detailed results in the section [https://wiki.mozilla.org/QA/Firefox3.5/TestPlan/Beta4#Detailed_Results below].
If you are running a BFT or FFT, Please record your detailed results in the section [https://wiki.mozilla.org/QA/Firefox3.5/TestPlan/RC1#Detailed_Results below].


=== Updates Testing ===
=== Updates Testing ===
We will be doing updates of all Alpha/Beta users to beta4.  We will be checking complete updates, and failover scenarios.   
We will be doing updates of all Alpha/Beta users to RC1.  We will be checking complete updates, and failover scenarios.   


Will be available on ''betatest'', ''releasetest'', and ''beta'' channels.
Will be available on ''betatest'', ''releasetest'', and ''beta'' channels.


Testcases and Results are tracked [https://wiki.mozilla.org/QA/Firefox3.5/TestPlan/Beta4/Updates here].
Testcases and Results are tracked [https://wiki.mozilla.org/QA/Firefox3.5/TestPlan/RC1/Updates here].


=== L10N ===
=== L10N ===
Visual inspection: Go through the P1 list of locales, and run a spotcheck to see if everything visually looks fine.  Refer to Fx3.1 [https://wiki.mozilla.org/L10n:Firefox_web_services_status L10N Requirements] for more specifics.
Visual inspection: Go through the P1 list of locales, and run a spotcheck to see if everything visually looks fine.  Refer to Fx3.1 [https://wiki.mozilla.org/L10n:Firefox_web_services_status L10N Requirements] for more specifics.


Testcases and Results are tracked [http://wiki.mozilla.org/QA/Firefox3.5/TestPlan/Beta4/L10N here].
Testcases and Results are tracked [http://wiki.mozilla.org/QA/Firefox3.5/TestPlan/RC1/L10N here].


=== Top Site Testing ===
=== Top Site Testing ===
* Tests on all platforms pass without a crash on the top-500 Sites
* Tests on all platforms result without a crash on the top-500 Sites
* Started a more complex run with more pages
* Started a more complex run with more pages


Line 135: Line 135:


=== Additional Resource Testing ===
=== Additional Resource Testing ===
Given the minimized focus testing around beta 4 due to time constraints, there are a few regression areas that will need some focus testing.  This area will call for additional resources to help QA in this effort to ship beta 4 in a timely manner.
Given the minimized focus testing around RC1 due to time constraints, there are a few regression areas that will need some focus testing.  This area will call for additional resources to help QA in this effort to ship RC1 in a timely manner.


* {{Bug|488015}}, {{bug|452498}} - Test around sites and content that this javascript fix may touch
* {{Bug|488015}}, {{bug|452498}} - Test around sites and content that this javascript fix may touch
Line 149: Line 149:


=== Fixed Bugs Verifications ===
=== Fixed Bugs Verifications ===
The following bugs are fixed for beta 4 and need verifications:
The following bugs are fixed for RC1 and need verifications:
* All Firefox/Platform 1.9.1 bugs: [http://tinyurl.com/8guodt http://tinyurl.com/8guodt]
* All Firefox/Platform 1.9.1 bugs: [http://tinyurl.com/8guodt http://tinyurl.com/8guodt]


=== Testday ===
=== Testday ===
A beta4 testday will be held and call for the community support to assist with testing.  Activities include executing litmus testcases, bug verifications, and focus on the new feature areas that are being implemented.  More can be found on [http://quality.mozilla.org/events/test-days QMO].
A RC1 testday will be held and call for the community support to assist with testing.  Activities include executing litmus testcases, bug verifications, and focus on the new feature areas that are being implemented.  More can be found on [http://quality.mozilla.org/events/test-days QMO].


= <FONT COLOR = "RED">Detailed Results </FONT> =
= <FONT COLOR = "RED">Detailed Results </FONT> =


[[Category:Shiretoko beta|4test]]
== Smoketests Results ==
== Smoketests Results ==
=== Windows XP [Henrik/Murali/Timr] ===
=== Windows XP [tester] ===
* Status: <b>RESULT</b> 4/24/09
=== Mac OS X [tester] ===
* Build: Gecko/20090423 Firefox/3.5b4 ID:20090423204732
=== Linux [tester] ===
* Bugs Found:
** [hskupin] New: {{Bug|490009}} - Migration wizard isn't launched after initial installation of Firefox
** Found unclear tests in Forms Manager
* Timr tested signed Win32 builds on XP
** Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
** Several unclear test cases: bugzilla does not store cookies according to site info dialog - fix test case?
** But b4 is '''PASS'''


=== Mac OSX [Aakash] ===
* Status: <b>PASS</b> 4/24/09
* Build: Gecko/20090423 Firefox/3.5b4 ID:20090423191946
* Bugs Found:
** Basic Printing test case and Installation are left
** [aakashd] Re-confirmed {{bug|475066}} - dragging a tab out of the browser window doesn't detach the tab
** [timr] Hit {{bug|461355}} - Crash in [HPSmartPrint@0xe1ef] after opening print dialog and canceling


=== Linux FC7 [Tracy] ===
== BFT Results ==
* Status - '''PASS''' 4/24/09
=== Windows XP [tester] ===
* Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
=== Mac OS X [tester] ===
** [tracy] Discovered {{bug|489973}} Double confirmation dialog for extension uninstall.
=== Linux [tester] ===
** [aakashd] Still seeing bug {{bug|485440}} - Password Manager dialog window is not resized on Esc Press
** [aakashd] Confirmed {{bug|426875}} - Enable sorting in the 'Page Info' => 'Media' table


== BFT Results ==
=== Windows XP [henrik] ===
* Build ID: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
* Status: '''PASS'''
* Issues Noted:
** [hskupin] New: {{Bug|490046}} - Geolocation fails when link is dragged into another tab with "update is null"
** [hskupin] {{Bug|469176}} - Attempting to drag a tab onto another application creates a new window instead
** [hskupin] {{Bug|366877}} - generated microsummaries updated against page loaded into tab even if page is stale
** [hskupin] {{Bug|433748}} - XML feed not populated after adding as livemark (Error: this._processor is null)
** [hskupin] {{Bug|127246}} Date() gives wrong timezone after time zone change
** [hskupin] New: {{Bug|490198}} - Import dialog isn't completely disabled while staying in Private Browsing mode
** [hskupin] New: {{Bug|490199}} - Selected option in new privacy pane not sticky when switching from never remember to customized view


=== Mac OS X [marcia/aakash/tchung] ===
== FFT Results ==
*Status: '''PASS'''
=== Windows XP [tester] ===
*Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
=== Mac OS X [tester] ===
*Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
=== Linux [tester] ===
*Status
*Existing Bugs Seen:
**[https://bugzilla.mozilla.org/show_bug.cgi?id=428602 Bug 42602]
*Issues Noted:
**http://bcheck.scanit.be/bcheck/index.php show an Exception in the Error Console
**Idle Flash player crash running 10.0.r22. Checked B3 and it shows there as well.
** [aakash] filed {{bug|490088}} - Search option for highlighted text is selectable in image context menu
** [aakash] Confirmed {{bug|175635}} - Empty default toolbars do not disappear automatically
** [tchung] Opened {{bug|490225}} - CMD-T opens a new window instead of a new tab when Downloads window is in focus


=== Linux [FC7,Tracy/Ubuntu,Aakash] ===
* Status - '''PASS'''
* Build Id: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4 ID:20090423192432
*
* Bugs Found:
** [aakashd] Re-confirmed STR for {{bug|485125}} -  Crash on close [@ libflashplayer.so@0x4110d ]


== Focused Testing ==_bug.cgi?id=490330 Bug 490330]
== Focused Testing ==_bug.cgi?id=490330 Bug 490330]

Revision as of 20:55, 19 May 2009

« QA/Firefox3.5/TestPlan

Firefox 3.5 RC1 Test Results

The covered testing for this release will focus on smoketests, bfts, l10n, top plugins, top financial sites, updates, and spotchecks.

Related Pages

Firefox 3.5 Tracking Site

Project contacts

PM: Mike Beltzner (beltzner)
Build: Nick Thomas (nthomas)
QA: Tony Chung (tester)

Schedule

Code Freeze
4/22?
Builds hand to QA
4/23?
QA test signoff
4/27?
  • See Test Coverage below

Build Location and IDs

Download:

Builds:

  • Mac: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
  • Linux: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4

Test Coverage

  • Areas Covered:
    • Smoketests
    • BFTs
    • L10n
    • Updates
    • Spotchecks & Webpages
    • Bug Fixes
  • Areas Not Covered:
    • FFTs
    • Topsites (covered in nightly automation)
    • Top Extensions
    • Top Themes
    • Top Plugins
    • Top Financial Sites
    • Accessibility

Test Results

Types of Tests Windows XP Mac OSX Linux
Smoketests tester - result (unsigned)
Timr - result (signed) 		Aakash/Timr - result tester - result Though randomly crashed a couple times.
BFTs tester/Murali - result tester/tester - result tester/tester - result
L10n tester - result tester - result tester - result
Updates tester
Betatest - result
releasetest - result
Beta - result 		tester
Betatest - result
releasetest - result
Beta - result 		tester
Betatest - result
releasetest - result
Beta - result
Additional Focused Testing QA team - result QA team - result QA team - result
Topsite Testing tester - result tester - result tester - result
Staging/Live Webpages & Spotchecks & Webpages tester - result tester - result tester - result

Smoketest Testing

Lite verification tests (or smoketests) will be first ran when the builds are ready. Tests are available on litmus under the Firefox 3.5 test suite. In any case the Smoketests result, we will raise blockers asap before proceeding with the bfts.

Please record your detailed results in the section below.

BFT / FFT Testing

BFTs are a more comprehensive set of litmus tests that cover the build testing after smoketests result. Tests are available on litmus under the Firefox 3.1 test suite.

FFTs are a full comprehensive set of litmus tests that extend coverage in areas that BFTs dont often get to. These are run at least once during the major milestone timeline.

If you are running a BFT or FFT, Please record your detailed results in the section below.

Updates Testing

We will be doing updates of all Alpha/Beta users to RC1. We will be checking complete updates, and failover scenarios.

Will be available on betatest, releasetest, and beta channels.

Testcases and Results are tracked here.

L10N

Visual inspection: Go through the P1 list of locales, and run a spotcheck to see if everything visually looks fine. Refer to Fx3.1 L10N Requirements for more specifics.

Testcases and Results are tracked here.

Top Site Testing

  • Tests on all platforms result without a crash on the top-500 Sites
  • Started a more complex run with more pages


Staging/Live Spotchecks & Webpages

1) Verify the following webpages that gets pushed to staging and then to the live sites. Look for content messaging, spelling mistakes, broken links & imnages, etc..

2) Check live site download links on all platforms. Please cover en-US, and 2 or 3 additional Tier 1 locales as well. Pay attention to the following areas:

  • Check for signed installer and applications (windows only)
  • Firstrun page, whatsnew page, Home page, Search plugin
  • Install, uninstall
  • General spotchecks and browser functionality
  • Confirm Build IDs

Additional Resource Testing

Given the minimized focus testing around RC1 due to time constraints, there are a few regression areas that will need some focus testing. This area will call for additional resources to help QA in this effort to ship RC1 in a timely manner.

Fixed Bugs Verifications

The following bugs are fixed for RC1 and need verifications:

Testday

A RC1 testday will be held and call for the community support to assist with testing. Activities include executing litmus testcases, bug verifications, and focus on the new feature areas that are being implemented. More can be found on QMO.

Detailed Results

Smoketests Results

Windows XP [tester]

Mac OS X [tester]

Linux [tester]

BFT Results

Windows XP [tester]

Mac OS X [tester]

Linux [tester]

FFT Results

Windows XP [tester]

Mac OS X [tester]

Linux [tester]

== Focused Testing ==_bug.cgi?id=490330 Bug 490330]

Testday Results

  • RC1 Testday scheduled may 29th
Retrieved from "https://wiki.allizom.org/index.php?title=QA/Firefox3.5/TestPlan/RC1&oldid=144829"