MozillaWiki

QA/Firefox3.5/TestPlan/Beta4: Difference between revisions

< QA‎ | Firefox3.5‎ | TestPlan
m (New page: <small>« QA/Firefox3.1/TestPlan</small> = Firefox 3.1 beta3 Test Results = The covered testing for this release will focus on smoketests, bfts, l10n, top...)
 
 
(104 intermediate revisions by 9 users not shown)
Line 1: Line 1:
<small>[[QA/Firefox3.1/TestPlan|&laquo; QA/Firefox3.1/TestPlan]]</small>
<small>[[QA/Firefox3.5/TestPlan|&laquo; QA/Firefox3.5/TestPlan]]</small>


= Firefox 3.1 beta3 Test Results =
= Firefox 3.5 beta4 Test Results =


The covered testing for this release will focus on smoketests, bfts, l10n, top plugins, top financial sites, updates, and spotchecks.   
The covered testing for this release will focus on smoketests, bfts, l10n, top plugins, top financial sites, updates, and spotchecks.   


== Related Pages ==
== Related Pages ==
[http://wiki.mozilla.org/Firefox3.1 Firefox 3.1 Tracking Site]
[http://wiki.mozilla.org/Firefox3.5 Firefox 3.5 Tracking Site]
* [https://wiki.mozilla.org/Releases/Firefox_3.1b3 Release Schedule]
* [https://wiki.mozilla.org/Releases/Firefox_3.5b4 Release Schedule]
* [https://wiki.mozilla.org/Firefox3.1/Features beta 3 features]
* [https://wiki.mozilla.org/Firefox3.5/Features beta 4 features]


== Project contacts ==  
== Project contacts ==  
Line 16: Line 16:


== Schedule ==  
== Schedule ==  
;Code Freeze: 3/3
;Code Freeze: 4/22?
;Builds hand to QA: 3/5
;Builds hand to QA: 4/23?
;QA test signoff: 3/12
;QA test signoff: 4/27?
* See Test Coverage below
* See Test Coverage below


== Build Location and IDs ==
== Build Location and IDs ==
Download:  
Download:  
* Build 1: ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/3.1b3-candidates/build1/
* Build 1: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/3.5b4-candidates/build1/
* Build 2: ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/3.1b3-candidates/build2/


Builds:
Builds:
* Build 1:
* Mac: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
** Mac 10.5: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3) Gecko/20090304 Firefox/3.1b3
* Linux: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
** Mac 10.4: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.1b3) Gecko/20090304 Firefox/3.1b3
** Windows: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b3) Gecko/20090304 Firefox/3.1b3
** Linux: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b3) Gecko/20090304 Firefox/3.1b3 (20090304215011)
 
* Build 2:
** Mac 10.5: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3
** Mac 10.4: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3
** Mac 10.5 PPC: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.5; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3
** Linux: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b3) Gecko/20090304 Firefox/3.1b3
** Windows XP: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3
** Windows 7: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3


== Test Coverage ==  
== Test Coverage ==  
Line 47: Line 34:
** BFTs
** BFTs
** L10n
** L10n
** Top Plugins
** Top Financial Sites
** Updates
** Updates
** Spotchecks & Webpages
** Spotchecks & Webpages
Line 56: Line 41:
** FFTs  
** FFTs  
** Topsites (covered in nightly automation)
** Topsites (covered in nightly automation)
** Top Extensions
** Top Themes
** Top Themes
** Top Plugins
** Top Financial Sites
** Accessibility
** Accessibility


== Test Results ==
== Test Results ==
{| class="fullwidth-table"
{| class="fullwidth-table"
! Types of Tests
! Types of Tests
Line 68: Line 55:
|-
|-
| Smoketests
| Smoketests
| Henrik - '''PASS''' <BR>Aakash (Vista) - '''PASS'''
| Henrik - '''PASS''' (unsigned) <br> Timr - '''PASS''' (signed)
| Aakash - '''PASS'''<BR>Tracy (PPC) - '''PASS'''
| Aakash/Timr - '''PASS'''
| juanb - '''PASS'''  
| Tracy - '''PASS''' <small>Though randomly crashed a couple times.</small>
|-
|-
| BFTs
| BFTs
| Aakash/Henrik - '''PASS'''
| Henrik/Murali - '''PASS'''  
| marcia - '''PASS'''
| Marcia/Tchung - '''PASS'''
| juanb - '''PASS'''
| Aakashd/Tracy - '''PASS'''  
|-
| FFTs
| Aakash/Henrik - '''result'''
| Tchung/marcia - '''result'''
| Aakash/TBD - '''result'''
|-
|-
| L10n
| L10n
| Tony - '''PASS'''
| Tchung - '''PASS'''
| Tony - '''PASS'''
| Tchung - '''PASS'''
| Tony - '''PASS'''
| Tchung - '''PASS'''
|-
|-
| Top Plugins
| Updates
| Aakash - '''PASS'''
| Tracy<BR> '''Betatest - PASS''' <BR> '''releasetest - PASS''' <BR> '''Beta - result'''  
| Murali - '''MIXED'''
| Henrik/Murali<BR> '''Betatest - PASS''' <BR> '''releasetest - PASS''' <BR> '''Beta - PASS'''  
| Murali - '''MIXED'''
| Aakashd/tester? <BR> '''Betatest - PASS''' <BR> '''releasetest - PASS''' <BR> '''Beta - PASS'''
|-
|-
| Top Financial sites
| Additional Focused Testing
| Tony - '''PASS'''
| Fx-team, QA team - '''PASS'''
| Marcia - '''PASS'''
| Fx-team, QA team - '''PASS'''
| Tony - '''PASS'''
| Fx-team, QA team - '''PASS'''
|-
|-
| Updates
| Topsite Testing
| Henrik<BR> '''Betatest - PASS''' <BR> '''releasetest - PASS''' <BR> '''Beta - PASS'''  
| Tomcat - '''PASS'''
| Murali<BR> '''Betatest - PASS''' <BR> '''releasetest - result''' <BR> '''Beta - result'''  
| Tomcat - '''PASS'''
| Aakash<BR> '''Betatest - PASS''' <BR> '''releasetest - PASS''' <BR> '''Beta - PASS'''
| Tomcat - '''PASS'''
|-
|-
| Staging/Live Webpages & Spotchecks & Webpages
| Staging/Live Webpages & Spotchecks & Webpages
| <web tester>, Henrik - '''PASS'''
| <web tester>, henrik - '''result'''
| <web tester>, Aakash - '''PASS'''
| <web tester>, marcia - '''result'''
| <web tester>, Tony - '''result'''
| <web tester>, Aakashd - '''PASS'''
|}
|}


Line 111: Line 93:
Lite verification tests (or smoketests) will be first ran when the builds are ready.  Tests are available on litmus under the [https://litmus.mozilla.org/run_tests.cgi?test_run_id=31 Firefox 3.1 test suite].  In any case the Smoketests result, we will raise blockers asap before proceeding with the bfts.
Lite verification tests (or smoketests) will be first ran when the builds are ready.  Tests are available on litmus under the [https://litmus.mozilla.org/run_tests.cgi?test_run_id=31 Firefox 3.1 test suite].  In any case the Smoketests result, we will raise blockers asap before proceeding with the bfts.


Please record your detailed results in the section [https://wiki.mozilla.org/QA/Firefox3.1/TestPlan/Beta3#Detailed_Results below].
Please record your detailed results in the section [https://wiki.mozilla.org/QA/Firefox3.5/TestPlan/Beta4#Detailed_Results below].
 
'''Note''' - please also make a smoketest pass against Windows 7 Beta, Mac OSX on PPC, and Windows Vista


=== BFT / FFT Testing ===
=== BFT / FFT Testing ===
Line 120: Line 100:
FFTs are a full comprehensive set of litmus tests that extend coverage in areas that BFTs dont often get to.  These are run at least once during the major milestone timeline.   
FFTs are a full comprehensive set of litmus tests that extend coverage in areas that BFTs dont often get to.  These are run at least once during the major milestone timeline.   


If you are running a BFT or FFT, Please record your detailed results in the section [https://wiki.mozilla.org/QA/Firefox3.1/TestPlan/Beta3#Detailed_Results below].
If you are running a BFT or FFT, Please record your detailed results in the section [https://wiki.mozilla.org/QA/Firefox3.5/TestPlan/Beta4#Detailed_Results below].


=== Updates Testing ===
=== Updates Testing ===
We will be doing updates of all Alpha users to beta3.  We will be checking complete updates, and failover scenarios.   
We will be doing updates of all Alpha/Beta users to beta4.  We will be checking complete updates, and failover scenarios.   


Will be available on ''betatest'', ''releasetest'', and ''beta'' channels.
Will be available on ''betatest'', ''releasetest'', and ''beta'' channels.


Testcases and Results are tracked [https://wiki.mozilla.org/QA/Firefox3.1/TestPlan/Beta3/Updates here].
Testcases and Results are tracked [https://wiki.mozilla.org/QA/Firefox3.5/TestPlan/Beta4/Updates here].


=== L10N ===
=== L10N ===
Visual inspection: Go through the P1 list of locales, and run a spotcheck to see if everything visually looks fine.  Refer to Fx3.1 [https://wiki.mozilla.org/L10n:Firefox_web_services_status L10N Requirements] for more specifics.
Visual inspection: Go through the P1 list of locales, and run a spotcheck to see if everything visually looks fine.  Refer to Fx3.1 [https://wiki.mozilla.org/L10n:Firefox_web_services_status L10N Requirements] for more specifics.


Testcases and Results are tracked [http://wiki.mozilla.org/QA/Firefox3.1/TestPlan/Beta3/L10N here].
Testcases and Results are tracked [http://wiki.mozilla.org/QA/Firefox3.5/TestPlan/Beta4/L10N here].


=== Top Plugins Testing ===
=== Top Site Testing ===
Select a few top plugins from AMO and perform the following testcases.  (examples: Flash, Acrobat, Java, Real Player)
* Tests on all platforms pass without a crash on the top-500 Sites
* Started a more complex run with more pages


Testcases:
* Enabling extensions and confirming web compatibility
* Disabling or Uninstalling plugins and confirming proper messaging and behavior
* Switch profiles and verify plugins are still working properly
Testcases and Results are tracked [http://wiki.mozilla.org/QA/Firefox3.1/TestPlan/Beta3/Plugins here].
=== Top Financial Sites Testing ===
* Select a few top [https://wiki.mozilla.org/QA:Website_Testing/Banks Banking sites]
* Select a few top [https://wiki.mozilla.org/QA:Website_Testing/FinancialServices Financial sites]
Testcases:
* Login/logout, verify functionality
* Look for any layout discrepencies
* Make a financial transaction if possible
* Be sure to test in private browsing mode also


=== Staging/Live Spotchecks & Webpages ===
=== Staging/Live Spotchecks & Webpages ===
Line 158: Line 123:
1) Verify the following webpages that gets pushed to staging and then to the live sites.  Look for content messaging, spelling mistakes, broken links & imnages, etc..
1) Verify the following webpages that gets pushed to staging and then to the live sites.  Look for content messaging, spelling mistakes, broken links & imnages, etc..


* Firstrun page ([https://www-trunk.stage.mozilla.com/en-US/firefox/3.1b3/firstrun/ Staging link])
* Firstrun page ([https://www-trunk.stage.mozilla.com/en-US/firefox/3.5b4/firstrun/ Staging link])
* Whatsnew Page ([https://www-trunk.stage.mozilla.com/en-US/firefox/3.1b3/whatsnew/ Staging link])
* Whatsnew Page ([https://www-trunk.stage.mozilla.com/en-US/firefox/3.5b4/whatsnew/ Staging link])
* Releasenotes page ([https://www-trunk.stage.mozilla.com/en-US/firefox/3.1b3/releasenotes/ Staging link])
* Releasenotes page ([https://www-trunk.stage.mozilla.com/en-US/firefox/3.5b4/releasenotes/ Staging link])


2) Check live site download links on all platforms.  Please cover en-US, and 2 or 3 additional Tier 1 locales as well.  Pay attention to the following areas:
2) Check live site download links on all platforms.  Please cover en-US, and 2 or 3 additional Tier 1 locales as well.  Pay attention to the following areas:
Line 168: Line 133:
* General spotchecks and browser functionality
* General spotchecks and browser functionality
* Confirm Build IDs
* Confirm Build IDs
=== Additional Resource Testing ===
Given the minimized focus testing around beta 4 due to time constraints, there are a few regression areas that will need some focus testing.  This area will call for additional resources to help QA in this effort to ship beta 4 in a timely manner.
* {{Bug|488015}}, {{bug|452498}} - Test around sites and content that this javascript fix may touch
* {{bug|475864}} - Test around video controls and file input handling
* {{bug|488497}} - Test around Geolocation work
* {{bug|488015}}, {{bug|470487}} - Test around crash fixes that need better repro steps
* {{bug|475066}} - Test around tearing away tabs
* {{bug|486236}} - Test around Ts performance
* {{bug|394759}} - Test around Session Restore / undo close window feature
* {{bug|480169}} - Test around new Clear Recent History (sprint) feature
* {{bug|422526}} - Test around localstorage (web developers)
* [http://bit.ly/FixedP1P2 Fixed P1,P2 1.9.1 bugs] from Apri 14th, need to be verified


=== Fixed Bugs Verifications ===
=== Fixed Bugs Verifications ===
The following bugs are fixed for beta 3 and need verifications:
The following bugs are fixed for beta 4 and need verifications:
* Firefox/Platform 1.9.1 bugs: [http://tinyurl.com/8guodt http://tinyurl.com/8guodt]
* All Firefox/Platform 1.9.1 bugs: [http://tinyurl.com/8guodt http://tinyurl.com/8guodt]


=== Testday ===
=== Testday ===
A beta3 testday will be held and call for the community support to assist with testing.  Activities include executing litmus testcases, bug verifications, and focus on the new feature areas that are being implemented.  More can be found on [http://quality.mozilla.org/events/test-days QMO].
A beta4 testday will be held and call for the community support to assist with testing.  Activities include executing litmus testcases, bug verifications, and focus on the new feature areas that are being implemented.  More can be found on [http://quality.mozilla.org/events/test-days QMO].


= <FONT COLOR = "RED">Detailed Results </FONT> =
= <FONT COLOR = "RED">Detailed Results </FONT> =
[[Category:Shiretoko beta|4test]]
== Smoketests Results ==
== Smoketests Results ==
=== Windows XP [Henrik] ===
=== Windows XP [Henrik/Murali/Timr] ===
* Build: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b3) Gecko/20090304 Firefox/3.1b3
* Status: <b>RESULT</b> 4/24/09
* Results
* Build: Gecko/20090423 Firefox/3.5b4 ID:20090423204732
** Build 1 (20090304233338) '''PASS''' - Results in Litmus (Complete)
* Bugs Found:
** Build 2 (20090305152042) '''PASS''' - Results in Litmus (Install/Uninstall)
** [hskupin] New: {{Bug|490009}} - Migration wizard isn't launched after initial installation of Firefox
** Found unclear tests in Forms Manager
* Timr tested signed Win32 builds on XP
** Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
** Several unclear test cases: bugzilla does not store cookies according to site info dialog - fix test case?
** But b4 is '''PASS'''


* Notes:
=== Mac OSX [Aakash] ===
** [Henrik]: filed [https://bugzilla.mozilla.org/show_bug.cgi?id=481714 bug 481714] - Switching to "Always ask" for file types handled by plugins don't work
* Status: <b>PASS</b> 4/24/09
* Build: Gecko/20090423 Firefox/3.5b4 ID:20090423191946
* Bugs Found:
** Basic Printing test case and Installation are left
** [aakashd] Re-confirmed {{bug|475066}} - dragging a tab out of the browser window doesn't detach the tab
** [timr] Hit {{bug|461355}} - Crash in [HPSmartPrint@0xe1ef] after opening print dialog and canceling


=== Windows Vista [Aakash] ===
=== Linux FC7 [Tracy] ===
* Build: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3
* Status - '''PASS''' 4/24/09
**20090305152042
* Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
** [tracy] Discovered {{bug|489973}} Double confirmation dialog for extension uninstall.
** [aakashd] Still seeing bug {{bug|485440}} - Password Manager dialog window is not resized on Esc Press
** [aakashd] Confirmed {{bug|426875}} - Enable sorting in the 'Page Info' => 'Media' table


*Notes:
== BFT Results ==
** Vista could not find network printers, so no test run there
=== Windows XP [henrik] ===
***Update: Marcia ran the test case from the Vista Lab PC and it passed
* Build ID: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
 
=== Windows 7 [Henrik] ===
* Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3
* Results
** Build 2 (20090305152042) '''PASS''' - Results in Litmus (Complete)
 
* Notes
** {{Bug|482004}} -  Form history not imported from Internet Explorer 8
 
=== Mac [Aakash] ===
 
* Build: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3) Gecko/20090304 Firefox/3.1b3
**20090304214853
**Status: <b>PASS</b>
* Build: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3
**20090305133223
**Status: <b>PASS</b>
 
* [tchung] Filed bug {{bug|481680}} - [Mac OSX] Beta 3 build 1 are showing "Mozilla Firefox3.1b3" in window title
** Blocker filed, fix will land in build 2 respin.
* [aakash] Filed bug {{bug|481694}} -  Home Page not imported from Safari
 
=== Mac PPC 10.4 [Tracy] ===
 
* Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3
* Status: '''PASS'''
* Status: '''PASS'''
* Issues Noted:
** [hskupin] New: {{Bug|490046}} - Geolocation fails when link is dragged into another tab with "update is null"
** [hskupin] {{Bug|469176}} - Attempting to drag a tab onto another application creates a new window instead
** [hskupin] {{Bug|366877}} - generated microsummaries updated against page loaded into tab even if page is stale
** [hskupin] {{Bug|433748}} - XML feed not populated after adding as livemark (Error: this._processor is null)
** [hskupin] {{Bug|127246}} Date() gives wrong timezone after time zone change
** [hskupin] New: {{Bug|490198}} - Import dialog isn't completely disabled while staying in Private Browsing mode
** [hskupin] New: {{Bug|490199}} - Selected option in new privacy pane not sticky when switching from never remember to customized view


=== Linux [Juan]===
=== Mac OS X [marcia/aakash/tchung] ===
* Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b3) Gecko/20090304 Firefox/3.1b3
*Status: '''PASS'''
** 20090304215011
*Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
* Status: '''PASS'''
*Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
* Bugs:
*Status
** Reopened (eventually re-Resolved) - {{Bug|475066}} - unable to detach tabs,
*Existing Bugs Seen:
** Opened new bug and nominated blocking 3.1 - {{Bug|481693}} - unable to detach tabs in Linux
**[https://bugzilla.mozilla.org/show_bug.cgi?id=428602 Bug 42602]
** Cannot (should not?) create web shortcuts. There is an existing bug, but I need to dig it out. It's also related to tabbed browsing (tear off).
*Issues Noted:
**http://bcheck.scanit.be/bcheck/index.php show an Exception in the Error Console
**Idle Flash player crash running 10.0.r22. Checked B3 and it shows there as well.
** [aakash] filed {{bug|490088}} - Search option for highlighted text is selectable in image context menu
** [aakash] Confirmed {{bug|175635}} - Empty default toolbars do not disappear automatically
** [tchung] Opened {{bug|490225}} - CMD-T opens a new window instead of a new tab when Downloads window is in focus


=== Spotchecks [marcia]===
=== Linux [FC7,Tracy/Ubuntu,Aakash] ===
*Filed {{bug|481687}} - Regression that affects only Mac
* Status - '''PASS'''
* Build Id: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4 ID:20090423192432
*
* Bugs Found:
** [aakashd] Re-confirmed STR for {{bug|485125}} - Crash on close [@ libflashplayer.so@0x4110d ]


== L10n testing ==
== Focused Testing ==
*marcia
**Regression tested Private Browsing, Clear Recent History and Privacy Pane Options, and Geolocation
***Could not reproduce [https://bugzilla.mozilla.org/show_bug.cgi?id=489885 Bug 489885] despite numerous tries.
***Filed [https://bugzilla.mozilla.org/show_bug.cgi?id=490330 Bug 490330]


* '''PASS'''
== Testday Results ==
* [tchung] {{Bug|475032}} - detected by hand that mac ko locale was missing from the /build2 directory.  Build Slave glitch, being manually rebuilt.
* Running Report can be found [https://litmus.mozilla.org/testday_report.cgi?testday_id=115&include_admin=on here] until its finalize at the end of the day
 
* Great Contributors outside MozQA: garrison, DOMi, zpao, FUX, Mavericks, adw, bloodsugarwilksm
== Financial Testing ==
 
*[tchung]
** Linux: Mozilla/5.0 (X11; U; Linux i686; fr; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3 '''PASS'''
** Windows: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3 '''PASS'''
** Ran testcases on ebay, amazon, emigrant, boa, fidelity, gmail, y!mail.
** No bugs found
 
== Plugins Testing ==
 
===Linux [Murali]===
* Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3
** 20090305134136
** Status: '''MIXED RESULT'''
 
*The Linux build has more leaks when firefox is run from commandline.
*Verified the JRE bug 474866 as fixed.
*Filed a bug 481963 which seems to be a DUP of 477110 [ Ref: Tomcat ]
*The following plugins fail to work on Linux
*Real Video crashes with Linux build. Known issue.
*Viewpoint Media player, IPIX are not tested. 
*Final result is PARTIAL
 
===  MAC OS X86 [Murali]===
*View Point Media Plugin : Failure
*IPIX : Not Applicable for MAC
*Rest of the plugins from testing page are good.
*Updated the results page for MAC as partial pass.
 
=== Win32 [ Aakash ] ===
 
* Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b3) Gecko/20090304 Firefox/3.1b3
**20090304233338
** Status: '''PASS'''
 
*Abode Reader: PASS
*Adobe Flash: PASS
*Java: PASS
*Quicktime: PASS, but no interaction with firefox on install/uninstall
*RealPlayer: PASS
*Shockwave: PASS
*WMP: PASS
*Silverlight: PASS
*VMP: NOT SUPPORTED ON 3.x
*iPIX: FAIL
** No real support for the plug-in since '06
** investigation ongoing: multiple sites do not respond to iPIX viewer enable/disable, but do respond to Java enable/disable.
** Used Plugins from http://www.ipix.com/download/alt_plugin.html and http://plugindoc.mozdev.org/windows-all.html#iPIX
* Divx: PASS
 
== BFT Results ==
=== Windows XP [Henrik/Aakash] ===
* Build 1: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b3) Gecko/20090304 Firefox/3.1b3
* Results
** Build 1 (20090304233338) '''PASS''' - Results in Litmus (Mostly all)
** Build 2 (20090305152042) '''PASS''' - Results in Litmus (Remaining + Tabs)
* Notes:
** [Henrik]: {{bug|463088}}, Tearing-off a tab to desktop shows denied cursor
** [Henrik]: Filed {{bug|481775}} Crash in [@ nsCookieService::RemoveCookieFromList(nsListIter&)]
** [Henrik] Sometimes after removing a ssl cert the page is still loaded and doesn't show the ssl exception page. No bug filed so far. Have to check why it happens.
** [Henrik] Leaving Private Browsing mode doesn't show up SSL sites only for each second time (not everytime) - commented in {{bug|463256}}
** [Henrik] {{bug|468700}} - Having "Ask me Everytime" enabled for Cookies during Private Browsing is unusable - asked for blocking1.9.1.
** [Henrik] Filed {{bug|481785}} -  No indication of 'Private Browsing' in window title when having enabled "Always on" Private Browsing Mode
** [Henrik] Filed {{bug|481786}} - Current session data is getting lost when starting Firefox in "Always On" mode
** [Henrik] Filed {{bug|481791}} - about:privatebrowsing page doesn't show a button when staying in private browsing mode
** [Henrik] Updated {{bug|134002}}, Content of plugins (flash, java applets) doesn't get rendered for printing or print preview
** [Henrik] {{bug|426260}}, Yahoo doesn't correctly decode the parameters served by the email handler
** [tchung] Reopened {{Bug|436686}}, Flash fails to play on windows 7, flash 10.0.22.87
** [aakash] Filed {{bug|481762}} - Clearing tag via typed-text does not un-check checkbox
** [aakash] Filed {{bug|481767}} - Tag field overlay issue when 2 or more bookmarks in the same tag
** [aakash] Filed {{bug|481904}} - Tab dragging in Scroll can not move freely from Left to Right
** [aakash] Added comments to {{bug|405511}} - Dragging a tab onto its content area reloads that page
 
=== Mac [Marcia] ===
* PASS
* Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3) Gecko/20090304 Firefox/3.1b3
* Flash version is Shockwave Flash 10.0 r12
* The Following Areas of BFT were run with Build 1:
** Migration
** Menu Bar
 
* Notes
** [Marcia/Henrik]: [https://bugzilla.mozilla.org/show_bug.cgi?id=476928 bug 476928] - Slow closing of tabs when entering Private Browsing mode
** [Henrik]: [https://bugzilla.mozilla.org/show_bug.cgi?id=473467 bug 473467] file picker dialog sometimes doesn't close when Open is clicked
** [marcia]: [https://bugzilla.mozilla.org/show_bug.cgi?id=481908 Bug 481908] Right clicking on video using Flash 10 causes screen to white out
** [marcia]: [https://bugzilla.mozilla.org/show_bug.cgi?id=481915 Bug 481915]  [Mac] View Certificate "General Tab" looks to be an active field that can be edited
** [marcia]: [https://bugzilla.mozilla.org/show_bug.cgi?id=481913 Bug 481913] In certain instances "View Cookies" button in Page Info does not open Cookies dialog
** [marcia]: [https://bugzilla.mozilla.org/show_bug.cgi?id=481951 Bug 481951] Print dialog very slow to open the first time
** [marcia]: [https://bugzilla.mozilla.org/show_bug.cgi?id=482286 Bug 482286] Cookie dialog does not display properly when visting cnn.com the first time
** [marcia]: [https://bugzilla.mozilla.org/show_bug.cgi?id=482512 Bug 482286] ESPN search plugin does not show correct favicon
 
* Areas for further investigation
**When you delete a home page and the field is blank, we default to the Firefox Start Page - is that correct?
**espn.go.com site - the Big Ipod Touch banner ad sometimes moves to the left when it is playing.
 
=== Linux [juanb]===
* Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3
** 20090305134136
** Status: '''Completed: PASS''' (as of Monday 6:30pm)
 
* Notes
** [Henrik]: Filed {{Bug|482232}} - Using search field in saved password window changes width of search field
** {{Bug|467565}} - Commented on existing bug regarding private browsing autostart preference and its behavior.
** https://litmus.mozilla.org/show_test.cgi?id=6290 - This test fails, but it also fails with 3.0.x and I need to investigate more.
** https://litmus.mozilla.org/show_test.cgi?id=5948 - There is no visual distinction between an open Folder and a closed one. Minor nit.
** I will investigate some of these and find out if there are actually problems. None of these would be blockers if they were confirmed.
 
== FFT Results ==
=== Windows [Aakash/Henrik] ===
* '''Build''': Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3 ID:20090305152042
 
* '''Results'''
** Installation - PASS (Aakash)
*** 4-5 Test cases had EULA excerpts removed and bugzilla links were revised for readability
** Migration - PASS (Aakash)
** Default Browser - In Progress (Aakash)
*** Vista specific
** Menu Bar - PASS (Aakash)
** Options - PASS (Aakash)
*** 25 Pass, 8 Unclear/Broken
** General - PASS (Aakash)
*** Found 2 Unclear/Broken test cases
** Tabbed Browsing - PASS (Aakash)
*** 37 Pass, 16 Unclear/Broken
** Search - PASS (Aakash)
*** 24 Pass, 2 Unclear/Broken
** Pop-Up and Annoying Blocking - PASS (Aakash)
*** 11 Pass, 2 Unclear/Broken
** History - PASS (Aakash)
*** 13 Pass, 1 Unclear/Broken
** MicroSummaries - PASS (Aakash)
** Software Update - PASS (Aakash)
 
=== Mac [Tchung/Marcia] ===
* tchung results
* 3.1b3 build 20090305133223
* Notes
** Filed {{Bug|483885}} - Import Dialog has unnecessary vert/hor scrollbars
 
* marcia's results
* Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3
* Bugs Filed
**{{Bug|484105}} - Error code: sec_error_bad_signature received during Litmus test run
**{{Bug|484112}} -  [JavaScript Error: "this._storage is null"] error after resetting Master Password
 
=== Linux [Aakash/TBD] ===
* Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3 ID:20090305134136
* Installation - PASS
** [aakashd] filed {{bug|484024}} - Dropping Tab onto Desktop doesn't Create a Web Shortcut
* Migration - PASS
* Menu Bar - PASS
* Options/Preferences - PASS
* General - PASS
* Location Bar & Autocomplete - PASS
 
== Websites Testing ==
* Notes
**[tchung] bug 482738 opened to track missing locales
** releasenotes link was pointing to 3.0.7 on the all-beta page
** Missing robot image on /firstrun , /whatsnew page
** [aakashd] filed {{bug|482987}} - Other Systems and Languages Hyperlink not Synced to Authstage page
 
[[Category:Shiretoko beta]]
 
== TestDay Results ==
* Bugs Verified
**
* Number of participants (37)
** Volunteer Mods (9):
*** Tomcat, aakashd, juanb, timr, tchung, ctalbert, tracy, whimboo, marcia, stephend
** Moz Employees (5):
*** dietrich, Jesse, abillings, jmaher, dolske
** Community Members (23):
*** DevilNut, sonny, Coce, JonW, RedHat71, MichaelKohler, GeekShadow, hanthana, Zero456, macgeek23, m-c, firey10may, kushalone, mzz, mtgordon, kbrosnan, boivie, othree, gstrickler, varzan, dustyrayburn, Andrei_Larin, Matti
* Test Cases Executed
** Total Number of Test Cases: 426
** 4 locales were represented
** Test Cases by platform
*** Windows: 225
*** Linux: 198
*** Mac: 3
** Top Tester: Zero456 with 73
** https://litmus.mozilla.org/testday_report.cgi?testday_id=114
 
== Bugs Filed Post Release from Feedback ==
*marcia
**[https://bugzilla.mozilla.org/show_bug.cgi?id=483644 Bug 483644] Regression from Firefox 3.0.7 - XML Parsing Error when loading this page (works in Firefox 3.0.7)
**[https://bugzilla.mozilla.org/show_bug.cgi?id=483270 Bug 483270] RadControls Rich Text Editor replaces certain characters with tilde

Latest revision as of 02:31, 28 April 2009

« QA/Firefox3.5/TestPlan

Firefox 3.5 beta4 Test Results

The covered testing for this release will focus on smoketests, bfts, l10n, top plugins, top financial sites, updates, and spotchecks.

Related Pages

Firefox 3.5 Tracking Site

Project contacts

PM: Mike Beltzner (beltzner)
Build: Nick Thomas (nthomas)
QA: Tony Chung (tchung)

Schedule

Code Freeze
4/22?
Builds hand to QA
4/23?
QA test signoff
4/27?
  • See Test Coverage below

Build Location and IDs

Download:

Builds:

  • Mac: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
  • Linux: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4

Test Coverage

  • Areas Covered:
    • Smoketests
    • BFTs
    • L10n
    • Updates
    • Spotchecks & Webpages
    • Bug Fixes
  • Areas Not Covered:
    • FFTs
    • Topsites (covered in nightly automation)
    • Top Extensions
    • Top Themes
    • Top Plugins
    • Top Financial Sites
    • Accessibility

Test Results

Types of Tests Windows XP Mac OSX Linux
Smoketests Henrik - PASS (unsigned)
Timr - PASS (signed) 		Aakash/Timr - PASS Tracy - PASS Though randomly crashed a couple times.
BFTs Henrik/Murali - PASS Marcia/Tchung - PASS Aakashd/Tracy - PASS
L10n Tchung - PASS Tchung - PASS Tchung - PASS
Updates Tracy
Betatest - PASS
releasetest - PASS
Beta - result 		Henrik/Murali
Betatest - PASS
releasetest - PASS
Beta - PASS 		Aakashd/tester?
Betatest - PASS
releasetest - PASS
Beta - PASS
Additional Focused Testing Fx-team, QA team - PASS Fx-team, QA team - PASS Fx-team, QA team - PASS
Topsite Testing Tomcat - PASS Tomcat - PASS Tomcat - PASS
Staging/Live Webpages & Spotchecks & Webpages <web tester>, henrik - result <web tester>, marcia - result <web tester>, Aakashd - PASS

Smoketest Testing

Lite verification tests (or smoketests) will be first ran when the builds are ready. Tests are available on litmus under the Firefox 3.1 test suite. In any case the Smoketests result, we will raise blockers asap before proceeding with the bfts.

Please record your detailed results in the section below.

BFT / FFT Testing

BFTs are a more comprehensive set of litmus tests that cover the build testing after smoketests result. Tests are available on litmus under the Firefox 3.1 test suite.

FFTs are a full comprehensive set of litmus tests that extend coverage in areas that BFTs dont often get to. These are run at least once during the major milestone timeline.

If you are running a BFT or FFT, Please record your detailed results in the section below.

Updates Testing

We will be doing updates of all Alpha/Beta users to beta4. We will be checking complete updates, and failover scenarios.

Will be available on betatest, releasetest, and beta channels.

Testcases and Results are tracked here.

L10N

Visual inspection: Go through the P1 list of locales, and run a spotcheck to see if everything visually looks fine. Refer to Fx3.1 L10N Requirements for more specifics.

Testcases and Results are tracked here.

Top Site Testing

  • Tests on all platforms pass without a crash on the top-500 Sites
  • Started a more complex run with more pages


Staging/Live Spotchecks & Webpages

1) Verify the following webpages that gets pushed to staging and then to the live sites. Look for content messaging, spelling mistakes, broken links & imnages, etc..

2) Check live site download links on all platforms. Please cover en-US, and 2 or 3 additional Tier 1 locales as well. Pay attention to the following areas:

  • Check for signed installer and applications (windows only)
  • Firstrun page, whatsnew page, Home page, Search plugin
  • Install, uninstall
  • General spotchecks and browser functionality
  • Confirm Build IDs

Additional Resource Testing

Given the minimized focus testing around beta 4 due to time constraints, there are a few regression areas that will need some focus testing. This area will call for additional resources to help QA in this effort to ship beta 4 in a timely manner.

Fixed Bugs Verifications

The following bugs are fixed for beta 4 and need verifications:

Testday

A beta4 testday will be held and call for the community support to assist with testing. Activities include executing litmus testcases, bug verifications, and focus on the new feature areas that are being implemented. More can be found on QMO.

Detailed Results

Smoketests Results

Windows XP [Henrik/Murali/Timr]

  • Status: RESULT 4/24/09
  • Build: Gecko/20090423 Firefox/3.5b4 ID:20090423204732
  • Bugs Found:
    • [hskupin] New: bug 490009 - Migration wizard isn't launched after initial installation of Firefox
    • Found unclear tests in Forms Manager
  • Timr tested signed Win32 builds on XP
    • Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
    • Several unclear test cases: bugzilla does not store cookies according to site info dialog - fix test case?
    • But b4 is PASS

Mac OSX [Aakash]

  • Status: PASS 4/24/09
  • Build: Gecko/20090423 Firefox/3.5b4 ID:20090423191946
  • Bugs Found:
    • Basic Printing test case and Installation are left
    • [aakashd] Re-confirmed bug 475066 - dragging a tab out of the browser window doesn't detach the tab
    • [timr] Hit bug 461355 - Crash in [HPSmartPrint@0xe1ef] after opening print dialog and canceling

Linux FC7 [Tracy]

  • Status - PASS 4/24/09
  • Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
    • [tracy] Discovered bug 489973 Double confirmation dialog for extension uninstall.
    • [aakashd] Still seeing bug bug 485440 - Password Manager dialog window is not resized on Esc Press
    • [aakashd] Confirmed bug 426875 - Enable sorting in the 'Page Info' => 'Media' table

BFT Results

Windows XP [henrik]

  • Build ID: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
  • Status: PASS
  • Issues Noted:
    • [hskupin] New: bug 490046 - Geolocation fails when link is dragged into another tab with "update is null"
    • [hskupin] bug 469176 - Attempting to drag a tab onto another application creates a new window instead
    • [hskupin] bug 366877 - generated microsummaries updated against page loaded into tab even if page is stale
    • [hskupin] bug 433748 - XML feed not populated after adding as livemark (Error: this._processor is null)
    • [hskupin] bug 127246 Date() gives wrong timezone after time zone change
    • [hskupin] New: bug 490198 - Import dialog isn't completely disabled while staying in Private Browsing mode
    • [hskupin] New: bug 490199 - Selected option in new privacy pane not sticky when switching from never remember to customized view

Mac OS X [marcia/aakash/tchung]

  • Status: PASS
  • Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
  • Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
  • Status
  • Existing Bugs Seen:
  • Issues Noted:
    • http://bcheck.scanit.be/bcheck/index.php show an Exception in the Error Console
    • Idle Flash player crash running 10.0.r22. Checked B3 and it shows there as well.
    • [aakash] filed bug 490088 - Search option for highlighted text is selectable in image context menu
    • [aakash] Confirmed bug 175635 - Empty default toolbars do not disappear automatically
    • [tchung] Opened bug 490225 - CMD-T opens a new window instead of a new tab when Downloads window is in focus

Linux [FC7,Tracy/Ubuntu,Aakash]

  • Status - PASS
  • Build Id: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4 ID:20090423192432
  • Bugs Found:
    • [aakashd] Re-confirmed STR for bug 485125 - Crash on close [@ libflashplayer.so@0x4110d ]

Focused Testing

  • marcia
    • Regression tested Private Browsing, Clear Recent History and Privacy Pane Options, and Geolocation

Testday Results

  • Running Report can be found here until its finalize at the end of the day
  • Great Contributors outside MozQA: garrison, DOMi, zpao, FUX, Mavericks, adw, bloodsugarwilksm
Retrieved from "https://wiki.allizom.org/index.php?title=QA/Firefox3.5/TestPlan/Beta4&oldid=141927"