Releases/Firefox 23/Test Plan

From MozillaWiki
Jump to navigation Jump to search

« Firefox 22 | Firefox 23 | Firefox 24 »

Firefox 23 Test Plan

Summary

Milestone Firefox 23
Status Currently in Aurora -- merging to Beta on 2013-06-24
Lead Anthony Hughes (irc: ashughes)
2nd Juan Becerra (irc: juanb)
Builds [ftp] [http] [web]

The following is the test plan for Firefox 23 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 22. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Schedule

  • [DONE] 2013.04.01: Nightly Merge
  • [DONE] 2013.05.13: Aurora Merge
  • [DONE] 2013.06.21: Pre-Beta feature sign-offs
  • [DONE] 2013.06.27: Beta 1
  • [DONE] 2013.07.02: Beta 2
  • [DONE] 2013.07.05: Beta 3
  • [DONE] 2013.07.09: Beta 4
  • [DONE] 2013.07.12: Beta 5
  • [DONE] 2013.07.16: Beta 6
  • [DONE] 2013.07.19: Beta 7
  • [DONE] 2013.07.23: Beta 8
  • 2013.07.26: Beta 9
  • 2013.07.31: Beta 10
  • 2013.08.05: Release Candidate
  • 2013.08.06: Final

Nightly

Merge

Functional Automation - result
Update Spotchecks - result

Features

See the Feature Release Tracking wiki for a general overview

Aurora

Migration Sign-off

Builds nightlies
Changeset tip (pushlog)
Checklist
  • [DONE] Automated functional tests
  • [DONE] Update tests on auroratest
  • [DONE] Update tests on aurora

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform Results [dashboard]
Windows XP [DONE] en-US de fr it: 0 fails
Windows Vista [DONE] en-US de fr it: 0 fails
Windows 7 32-bit [DONE] en-US de fr it: 0 fails
Windows 7 64-bit [DONE] en-US de fr it: 0 fails
Windows 8 32-bit [DONE] en-US de fr it: 0 fails
Windows 8 64-bit [DONE] en-US de fr it: 0 fails
Mac OS X 10.6 [DONE] en-US de fr it: 0 fails
Mac OS X 10.7 [DONE] en-US de fr it: 0 fails
Mac OS X 10.8 [DONE] en-US de fr it: 0 fails
Ubuntu 12.04 32-bit [DONE] en-US de fr it: 0 fails
Ubuntu 12.04 64-bit [DONE] en-US de fr it: 0 fails
Test Failures (known bugs)

Updates

Pre-migration Update Throttling
Platform Tested Version Target Version Result
Windows XP Firefox 21.0a2 en-US no update PASS
Firefox 22.0a2 ar no update PASS
Mac OS X 10.8 Firefox 20.0a2 en-US no update PASS
Firefox 22.0a2 ja-JP-mac no update PASS
Ubuntu 12.04 64-bit Firefox 19.0a2 en-US no update PASS
Firefox 22.0a2 de no update PASS
Auroratest
Platform Tested Version Target Version Result
Windows XP Firefox 22.0a2 2013-04-26 en-US Firefox 23.0a2 2013-05-15 en-US PASS
Firefox 22.0a2 2013-05-08 de Firefox 23.0a2 2013-05-15 de PASS
Mac OS X 10.8 Firefox 14.0a2 2012-06-02 en-US Firefox 23.0a2 2013-05-15 en-US PASS
Firefox 22.0a2 2013-05-10 fr Firefox 23.0a2 2013-05-15 fr PASS
Ubuntu 12.04 64-bit Firefox 21.0a2 2013-03-07 en-US Firefox 23.0a2 2013-05-15 en-US PASS
Firefox 22.0a2 2013-05-09 it Firefox 23.0a2 2013-05-15 it PASS
Issues
Aurora
Platform Tested Version Target Version Result
Windows XP Firefox 22.0a2 2013-04-26 en-US Firefox 23.0a2 2013-05-17 en-US PASS
Firefox 22.0a2 2013-05-08 de Firefox 23.0a2 2013-05-17 de PASS
Mac OS X 10.8 Firefox 14.0a2 2012-06-02 en-US Firefox 23.0a2 2013-05-17 en-US PASS
Firefox 22.0a2 2013-05-10 fr Firefox 23.0a2 2013-05-17 fr PASS
Ubuntu 12.04 64-bit Firefox 21.0a2 2013-03-07 en-US Firefox 23.0a2 2013-05-17 en-US PASS
Firefox 22.0a2 2013-05-09 it Firefox 23.0a2 2013-05-17 it PASS
Issues
Post-migration Update Unthrottling
Platform Tested Version Target Version Result
Windows XP Firefox 22.0a2 2013-04-26 en-US Firefox 23.0a2 2013-05-17 en-US PASS
Firefox 22.0a2 2013-05-08 de Firefox 23.0a2 2013-05-17 de PASS
Mac OS X 10.8 Firefox 14.0a2 2012-06-02 en-US Firefox 23.0a2 2013-05-17 en-US PASS
Firefox 22.0a2 2013-05-10 fr Firefox 23.0a2 2013-05-17 fr PASS
Ubuntu 12.04 64-bit Firefox 21.0a2 2013-03-07 en-US Firefox 23.0a2 2013-05-17 en-US PASS
Firefox 22.0a2 2013-05-09 it Firefox 23.0a2 2013-05-17 it PASS

Beta

Beta 1

Build Info

Builds FTP, HTTP
Build ID 20130625125232
Changes 77a8241ae55d
Configs mozmill

Checklist

Task Status
Feature Sign-off [DONE]
Automated Tests [DONE]
Bug Fix Verifications [DONE]
Update Tests (betatest) [DONE]
Update Tests (releasetest) [DONE]
Stability & Support sign-off [DONE]
Update Tests (beta) [ON TRACK]

Automated Tests

Platform Status (results)
Windows XP [DONE] 0 fails
Windows Vista [DONE] 0 fails
Windows 7 32-bit [DONE] 0 fails
Windows 7 64-bit [DONE] 0 fails
Windows 8 32-bit [DONE] 0 fails
Windows 8 64-bit [DONE] 3 fails
Mac OSX 10.6 [DONE] 0 fails
Mac OSX 10.7 [DONE] 0 fails
Mac OSX 10.8 [DONE] 0 fails
Ubuntu Linux 32-bit [DONE] 1 fails
Ubuntu Linux 64-bit [DONE] 0 fails

Failures (known)

Feature Sign Off

Feature QA Contact Status
Mixed content blocking Matt Wobensmith, Mihai Morar [DONE] Ready for Beta
Add click-to-start disabled plugins Paul Silaghi [DONE] Ready for Beta
SocialAPI Share Panel Anthony Hughes, Samvedana Gohil [DONE] Ready for Beta
SocialAPI SocialMark Anthony Hughes, Samvedana Gohil [DONE] Ready for Beta
SocialAPI Web Install Anthony Hughes, Samvedana Gohil [DONE] Ready for Beta
SocialAPI Fullscreen Video Chat Anthony Hughes, Samvedana Gohil [DONE] Ready for Beta
Consolidate search preferences Manuela Muntean [DONE] Ready for Beta
New scrollbar style on OSX 10.7+ Marc Schifer [DONE] Ready for Beta

Regression Tests

Moztrap Testrun #1519

Platform Status (results)
Windows 7 32-bit [DONE] 2 fails
Windows 8 64-bit [DONE] 1 fails
Mac OSX 10.8 [DONE] 0 fails
Ubuntu Linux 32-bit [DONE] 0 fails

Bug Fix Verifications

Bugs [more] Status
bug 881991 [FHR] Duplicate aborted sessions being inserted need-info
bug 881370 Crash when using JSD2 Debugger API for profiling [BLOCKED]
bug 880042 Multi-file form field nsIFilePicker crash if overriding component has not implemented a domfiles getter [DONE]
bug 878449 Crash in mozilla::widget::JumpListBuilder::AddListToBuild [DONE]
bug 877612 Cannot enter into fullscreen mode after print dialog is opened [DONE]
bug 877097 crash in mozilla::layout::ScrollbarActivity::UpdateOpacity REOPENED
bug 876856 Crash in mozilla::dom::MediaStreamList::Length after closing a peer connection [DONE]
bug 876721 crash in mozilla::gl::SharedSurface_ANGLEShareHandle::Create [DONE]
bug 874090 Crash in mozilla::dom::Notification::GetPermissionInternal [DONE]
bug 873963 Arbitrary code execution from Font Inspector [DONE]
bug 872758 [FHR] Multi delete documents [BLOCKED]
bug 871942 crash in nsWebShellWindow::RequestWindowClose [DONE]
bug 869825 crash in mozilla::dom::ReparentWrapper [DONE]
bug 868904 Too-much-recursion crash with CSS animated transform, SVG [DONE]
bug 868337 crash in js::ion::DoCallFallback [DONE]
bug 868257 watchPosition only sends one location need-info
bug 867445 Remove test pilot from Firefox [DONE]
bug 867226 Incomplete framebuffer abort in mozilla::layers::LayerManagerOGL::CreateFBOWithTexture [DONE]
bug 866339 crash in mozilla::dom::DocumentBinding::createDocumentFragment [DONE]
bug 865584 crash in js::ion::IonBuilder::createThisScriptedSingleton [DROPPED]
bug 865260 crash in js::IsStandardClassResolved [DONE]
bug 864505 [SocialAPI] active chat titlebar fails to change [DONE]
bug 863646 Start up crash in nsContentUtils::GetCurrentJSContext in profile manager [DONE]
bug 862610 Crash in mozilla::dom::HTMLImageElementBinding::DefineDOMInterface [DONE]
bug 862084 Crash in mozilla::dom::HTMLSelectElementBinding::item [DONE]
bug 860560 Search Box should respond to the changing of the selectedEngine/defaultenginename preferences [DONE]
bug 860149 crash in mozilla::image::Decoder::Write [DONE]
bug 859718 crash in mozilla::image::RasterImage::DecodePool::DecodeJob::Run [DONE]
bug 859714 File open dialog buttons not working [DONE]
bug 858946 Download panel doesn't display date correctly due to toLocaleFormat() [DONE]
bug 858800 crash in mozilla::plugins::MiniShmBase::GetWritePtr [DONE]
bug 855370 Remove the ability to not "Always show the tab bar" [DONE]
bug 854319 crash in mozilla::MediaResource::`.* deleting destructor [DONE]
bug 851934 crash in js::mjit::stubs::ValueToBoolean on Outlook.com [DONE]
bug 851702 Remove "Enable JavaScript" checkbox from Prefs [DONE]
bug 851701 Remove "Load images automatically" checkbox from Prefs [DONE]
bug 847267 Enable DXVA2 on Windows Vista+ to accelerate H.264 video decoding [DONE]
bug 839206 Replace plugin installation notification bar with door hanger [DONE]
bug 821073 full screen support for video chats [DONE]
bug 704063 Add unprefixed requestAnimationFrame [DONE]
bug 331105 Allow users to set a default search engine [DONE]
bug 52500 Make regular CSS properties apply on <input type='file'> [DONE]
Metrics: 36/42 verified (86%)

Update Tests

Platform Betatest [report] Releasetest [report] Beta [report]
Windows XP [DONE] 0 fails [DONE] 0 fails [DONE] 0 fails
Windows Vista [DONE] 0 fails [DONE] 0 fails [DONE] 0 fails
Windows 7 32-bit [DONE] 0 fails [DONE] 0 fails [DONE] 0 fails
Windows 7 64-bit [DONE] 0 fails [DONE] 0 fails [DONE] 0 fails
Windows 8 32-bit [DONE] 0 fails [DONE] 0 fails [DONE] 0 fails
Windows 8 64-bit [DONE] 0 fails [DONE] 0 fails [DONE] 0 fails
Mac OSX 10.6 [DONE] 0 fails [DONE] 0 fails [DONE] 0 fails
Mac OSX 10.7 [DONE] 0 fails [DONE] 0 fails [DONE] 0 fails
Mac OSX 10.8 [DONE] 0 fails [DONE] 0 fails [DONE] 0 fails
Ubuntu Linux 32-bit [DONE] 0 fails [DONE] 0 fails [DONE] 0 fails
Ubuntu Linux 64-bit [DONE] 0 fails [DONE] 0 fails [DONE] 0 fails

New Issues

Bugs Status
bug 887298 New created profile doesn't appear in the Profile Manager [NEW]
bug 887276 Firefox 23 icon is changed [DONE]
bug 887230 Add the ability to perform searches after a number from the URL bar [DUPE]
bug 887228 Sound stops when video is played at 0.25x rate [NEW]

Beta 2

Build Info

Testing Info

Test Result Criteria
Automated Tests PASS (details) All Mozmill ondemand_functional testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Locales: da, de, en-US, es-ES, fr, it, ja, ko, pl, pt-BR, ro, ru, zh-TW
Update Tests
(betatest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Locales: da, de, en-US, es-ES, fr, it, ja, ko, pl, pt-BR, ro, ru, zh-TW
  • Versions: 19.0b*, 20.0b*, 21.0b*, 22.0b* 23.0b1
Update Tests
(releasetest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Locales: da, de, en-US, es-ES, fr, it, ja, ko, pl, pt-BR, ro, ru, zh-TW
  • Versions: 19.0b*, 20.0b*, 21.0b*, 22.0b* 23.0b1
Update Tests
(beta) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Locales: da, de, en-US, es-ES, fr, it, ja, ko, pl, pt-BR, ro, ru, zh-TW
  • Versions: 19.0b*, 20.0b*, 21.0b*, 22.0b* 23.0b1
Regression Tests PASS (details) All tests in MozTrap Testrun #1638 pass without any serious regressions
  • Tests: Breakpad, MD5 hash algorithm, and Plug-in compatibility exploratory smoketests
  • Platforms: Windows 7 32-bit, Mac OSX 10.7
  • Locales: en-US
  • Versions: 23.0b2#1

Issues Found

  • bug 889519 [mozmill] Lots of ondemand_update failures with various Java IO Exceptions

Beta 3

Build Info

Testing Info

Test Result Criteria
Automated Tests PASS (details) All Mozmill ondemand_functional testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 23.0b3#1 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(betatest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 19.0b1-23.0b2 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(releasetest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 19.0b1-23.0b2 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(beta) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 19.0b1-23.0b2 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Regression Tests PASS (details) All Moztrap tests pass with no new regressions
  • Platforms: Windows 8.1 32-bit, Windows 8.1 64-bit, Mac OSX 10.9
  • Builds: 23.0b3#1 en-US

Issues Found

Full Query
ID Summary Status Resolution
890247 The Post Privacy Setting dropdown of the Share panel is not fully displayed RESOLVED WONTFIX
890258 Share login panel partly white RESOLVED WONTFIX
890260 Login errors in the Share panel are displayed partially over the sign in form RESOLVED WONTFIX
890264 An option for logging out of Facebook should be added to the Facebook Messenger menu RESOLVED WONTFIX
890283 browser.search.defaultenginename should not be case sensitive RESOLVED INVALID

5 Total; 0 Open (0%); 5 Resolved (100%); 0 Verified (0%);


Beta 4

Build Info

Testing Info

Test Result Criteria
Automated Tests PASS (details) All Mozmill ondemand_functional testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 23.0b4#1 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(betatest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 19.0b1-23.0b3 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(releasetest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 19.0b1-23.0b3 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(beta) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 19.0b1-23.0b3 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Regression Tests PASS (details) All Moztrap tests pass with no new regressions
  • Platforms: Windows Vista, Windows 7 32-bit, Mac OSX 10.6, Ubuntu Linux 32-bit
  • Builds: 23.0b4#1 en-US

Issues Found

Bugs Status
No bugs found...

Beta 5

Build Info

Testing Info

Test Result Criteria
Automated Tests PASS (details) All Mozmill ondemand_functional testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 23.0b5#1 da de en-US es-ES fr it ja ko pt-BR ro ru
Update Tests
(betatest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 19.0b1-23.0b4 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(releasetest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 19.0b1-23.0b4 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW
Update Tests
(beta) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 19.0b1-23.0b4 da de en-US es-ES fr it ja ko pl pt-BR ro ru zh-TW

Issues Found

Beta 6

Build Info

Testing Info

Test Result Criteria
Automated Tests PASS (details) All Mozmill ondemand_functional testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 23.0b6#1
  • Locales: af da de en-US es-ES es-MX fi fr hr hu it ja ko ku ml or pl pt-BR pt-PT ro ru sr zh-CN zh-TW zu
Update Tests
(betatest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 19.0b1-23.0b5
  • Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Update Tests
(releasetest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 19.0b1-23.0b5
  • Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Update Tests
(beta) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 19.0b1-23.0b5
  • Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Regression Tests PASS (details) All Moztrap tests pass with no new regressions
  • Platforms: Windows XP, Windows 7 64-bit, Mac OSX 10.8, Ubuntu Linux 32-bit
  • Builds: 19.0b1-23.0b5
  • Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW

Note: On Ubuntu 12.10 32bit, the Copy & Paste from plug-in content to another application, in the case of quitting Firefox before pasting, doesn't work (same behavior on Chromium and Opera)

Issues Found

  • bug 894397 Norton toolbar is not displayed in the Add-ons Manager
  • bug 894412 Text selected from Gmail is removed from clipboard after Firefox restart
  • bug 894509 [mozmill] l10n failures in testTabbedBrowser/testNewTab.js
  • bug 894510 [mozmill] Failure - domain name mismatch in testSearchSelection.js
  • bug 894513 [mozmill] Location bar should contain pasted clipboard content - got , expected ipsum in testPasteLocationBar.js

Beta 7

Build Info

Testing Info

Test Result Criteria
Automated Tests PASS (details) All Mozmill ondemand_functional testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 23.0b7#1
  • Locales: af da de en-US es-ES es-MX fi fr hr hu it ja ko ku ml or pl pt-BR pt-PT ro ru sr zh-CN zh-TW zu
Update Tests
(betatest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 19.0b1-23.0b6
  • Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Update Tests
(releasetest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 19.0b1-23.0b6
  • Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Update Tests
(beta) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 19.0b1-23.0b6
  • Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Regression Tests PASS (details) All Moztrap tests pass with no new regressions
  • Platforms: Windows XP, Windows 7 64-bit, Mac OSX 10.8, Ubuntu Linux 32-bit
  • Builds: 23.0b7#1
  • Locales: en-US

Issues Found

  • bug 856541 [mozmill] testEnableCookies.js fails with "Disconnect Error: Application unexpectedly closed"
  • bug 894509 [mozmill] testNewTab.js fails with "Correct tab title - 'Pestaña nueva' should equal 'Nueva pestaña'"
  • bug 894513 [mozmill] testPasteLocationbar.js fails with "Location bar should contain pasted clipboard content - got , expected ipsum"
  • bug 895996 [mozmill] testAddMozSearchProvider.js fails with "Disconnect Error: Application unexpectedly closed"
  • bug 835132 [mozmill] testSearchSelection.js fails with "Current domain name matches target domain name"
  • bug 818128 [mozmill] testVisibleItemsMax.js fails with "Number of visible rows should equal 6"

Beta 8

Build Info

Testing Info

Test Result Criteria
Automated Tests PASS (details) All Mozmill ondemand_functional testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 23.0b8#1
  • Locales: af da de en-US es-ES es-MX fi fr hr hu it ja ko ku ml or pl pt-BR pt-PT ro ru sr zh-CN zh-TW zu
Update Tests
(betatest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 19.0b1-23.0b7
  • Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Update Tests
(releasetest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 19.0b1-23.0b7
  • Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Update Tests
(beta) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 19.0b1-23.0b7
  • Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Regression Tests PASS (details) Regression testing around beta plugins/drivers
  • Platforms: Windows 8.1, Mac OSX 10.9
  • Builds: 23.0b8#1
  • Locales: en-US

Issues Found

  • [ON TRACK] bug 894513 [mozmill] Failure in testPasteLocationBar.js
  • [ON TRACK] bug 895996 [mozmill] Failure in testAddMozSearchProvider.js
  • [ON TRACK] bug 835132 [mozmill] Failure in testSearchSelection.js
  • [ON TRACK] bug 818128 [mozmill] Failure in testVisibleItemsMax.js
  • [ON TRACK] bug 866599 [mozmill] Failure in testPasswordSavedAndDeleted.js
  • [DROPPED] bug 897182 [mozmill] Failure in testDefaultBookmarks.js
  • [DROPPED] bug 897183 [mozmill] Failure in testPreferences_masterPassword
  • [DROPPED] bug 897177 [mozmill] Multiple failures with Firefox 23 en-US on Ubuntu

Beta 9

Build Info

Testing Info

Test Result Criteria
Automated Tests PASS (details) All Mozmill ondemand_functional testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 23.0b9#1
  • Locales: af da de en-US es-ES es-MX fi fr hr hu it ja ko ku ml or pl pt-BR pt-PT ro ru sr zh-CN zh-TW zu
Update Tests
(betatest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 19.0b1-23.0b8
  • Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Update Tests
(releasetest) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 19.0b1-23.0b8
  • Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Update Tests
(beta) 		PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 19.0b1-23.0b8
  • Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Regression Tests PASS (details) Regression testing around mixed content blocking (ex. apps.facebook.com, Koding.com Apps, Firstgroup.com Purchases)
  • Platforms: Windows XP, Windows 7 64-bit, Mac OSX 10.7, Ubuntu 13.04 32-bit
  • Builds: 23.0b9#1
  • Locales: en-US

Issues Found

  • FIXED [BLOCKED] bug 898587 No updates available to Firefox 23.0b9
  • FIXED [BLOCKED] bug 898574 Mozmill nodes in *.qa.scl3.mozilla.com cannot reach external machines via the proxy
  • bug 898391 Bubble Adventures game keeps on reloading
  • bug 898402 Inconsistent behavior on some of the Facebook apps
  • bug 898399 Doorhanger is displayed only at first loading of https://koding.com/ using new profile
  • bug 894513 [mozmill] testPasteLocationbar.js fails with "Location bar should contain pasted clipboard content - got , expected ipsum"
  • bug 895996 [mozmill] testAddMozSearchProvider.js fails with "Disconnect Error: Application unexpectedly closed"
  • bug 835132 [mozmill] testSearchSelection.js fails with "Current domain name matches target domain name"
  • bug 818128 [mozmill] testVisibleItemsMax.js fails with "Number of visible rows should equal 6"
  • bug 826693 [mozmill] testGreenLarry.js fails with "controller.waitForPageLoad(): Timeout waiting for page loaded."
  • bug 898568 [mozmill] testSearchSuggestions.js fails with "Suggestions Mozilla from Google and SAPO search providers are different - got 'false' "

Beta 10

Build Info

Testing Info

Test Result Criteria
Automated Tests PASS (details) All Mozmill ondemand_functional testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 23.0b10#1
  • Locales: af da de en-US es-ES es-MX fi fr hr hu it ja ko ku ml or pl pt-BR pt-PT ro ru sr zh-CN zh-TW zu
Update Tests (betatest) PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 19.0b1-23.0b9
  • Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Update Tests (releasetest) PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 19.0b1-23.0b9
  • Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Update Tests (beta) PASS (details) All Mozmill ondemand_update testruns PASS with no new failures
  • Platforms: Windows XP/Vista/7/8, Mac OSX 10.6/10.7/10.8, Ubuntu Linux 12.04/12.10
  • Builds: 19.0b1-23.0b9
  • Locales: da de en-US es-ES fi fr it ja ja-JP-mac ko pl pt-BR ro ru zh-TW
Regression Tests PASS (details) All Moztrap regression tests PASS with no regressions found

Issues Found

  • bug 894513 [mozmill] testPasteLocationbar.js fails with "Location bar should contain pasted clipboard content - got , expected ipsum"
  • bug 895996 [mozmill] testAddMozSearchProvider.js fails with "Disconnect Error: Application unexpectedly closed"
  • bug 835132 [mozmill] testSearchSelection.js fails with "Current domain name matches target domain name"
  • bug 818128 [mozmill] testVisibleItemsMax.js fails with "Number of visible rows should equal 6"
  • bug 899565 Linkedin search box slightly misplaced
  • bug 899579 Unable to select only a part of the link from html popup on Youtube HTML5 Videos

Release Candidate

Build Info

Build 1 Build 2
ID 20130730113002 20130801200732
Builds FTP, HTTP FTP, HTTP
Pushlog 2c423c4c6d0b 5efffaf39814
Configs ondemand.cfg ondemand.cfg

Feature Sign Off

Feature Owner Status
Mixed content blocking Matt Wobensmith DONE - Ready for Release
Share, SocialMark, WebInstall, Tear-off Chat Samvedana Gohil DONE - Ready for Release
Consolidated Search Preferences Matt Wobensmith DONE - Ready for Release
Max OSX Scrollbars Marc Schifer Delayed to Firefox 24

Regression Testing

Criteria Description Result Reports
Mozmill Tests Automated functional smoketests utilizing Mozmill PASS build1, build2
Moztrap Tests Manual tests covering areas at high risk to regression PASS build1
AMD Radeon Stability Tests Manual dogfooding to shake out intermittent instability PASS n/a
Partner Repack Update Tests Manual spotchecking of updates for partner repack builds PASS n/a

Update Testing

Criteria Description Result Reports
Betatest Automated software update tests on the betatest channel PASS build1, build2
Releasetest Automated software update tests on the releasetest channel PASS build1
Release Automated software update tests on the release channel PASS build1

Issues Found

Bug Status
bug 900520 Yahoo Repack Build - the installation can't be customized RESOLVED INVALID
bug 900516 Yahoo Repack Build - Installer keeps prompting to close Firefox before installing RESOLVED INVALID
bug 900463 Share panel on Facebook doesn't initially load content NEW
bug 900454 The Share button doesn't change colour to indicate the page has been shared RESOLVED WONTFIX
bug 900414 Dragged and dropped text from a HTML page to webmail is pasted in Chinese NEW
bug 900373 Callout missing on YouTube's Statistics NEW

Bug Triage

Query Description Remaining
QAWANTED Bugs requiring immediate investigative assistance 0
VERIFYME Bugs with fixes requiring immediate verification 15
FIXED Bugs with fixes which should be verified as soon as possible 146
REGRESSION Bugs requiring identification of the first affected build 222
UNCONFIRMED Bugs requiring confirmation if they are reproducible in a current build 73

Known Issues

Full Query
ID Summary Status Resolution
449392 Running the Java Applet Video Player together with scrolling the Site causes paint error RESOLVED INVALID
582097 Java error, ClassNotFoundException: Tcpbw100.class fails with 4.0b3, works w/ 4.0b1 RESOLVED WORKSFORME
693808 Entering numbers or single words and then pressing Enter in the location bar should bring search results VERIFIED FIXED
806116 touchpad TAP only triggers ":active" with double-tap on element with tabindex NEW
811085 Can't use Adobe Reader for PDF files RESOLVED WONTFIX
842566 EXC_BAD_ACCESS (SIGSEGV) after building with DMD NEW
881027 100% cpu when on new maps.google.com (may be unable to kill, requiring reboot) RESOLVED WORKSFORME
885351 Google is always used in about:home and the URL bar when setting the default search engine to an invalid one VERIFIED FIXED
887228 Sound stops when video is played at 0.25x rate RESOLVED WORKSFORME
887298 New created profile doesn't appear in the Profile Manager RESOLVED WORKSFORME
888885 Weird performance differences in new google maps RESOLVED INACTIVE
888914 Bottom of page hidden with Dev Tools open RESOLVED WONTFIX
889320 Firefox does not show PDF when Content Security Policy is enabled VERIFIED FIXED
889790 Need to be able to add Video to an existing audio-only webrtc call or remove Video RESOLVED FIXED
890355 Attachment P7M not showing NEW
890843 Missing control capabilities on animated GIFs for a user (GIF Player capabilities: pause, stopping, per frame forward etc) RESOLVED WONTFIX
891424 Modifying XUL panel while open causes the browser to hide VERIFIED FIXED
892375 DevTools Console Net doesn't allow opening URLs on first click RESOLVED WORKSFORME
892431 Cant leave a webpage UNCONFIRMED
892506 Autocomplete/Passwordmanager automatically fills in first field on website with a wrong value RESOLVED INVALID
894397 Norton toolbar is not displayed in the Add-ons Manager RESOLVED INVALID
894412 Text selected from Gmail is removed from clipboard after Firefox restart RESOLVED DUPLICATE
898399 Doorhanger is displayed only at first loading of https://koding.com/ using new profile RESOLVED INVALID
900373 Callout missing on YouTube's Statistics RESOLVED INVALID
900414 Dragged and dropped text from a HTML page to webmail is pasted in Chinese RESOLVED DUPLICATE
900463 Share panel on Facebook doesn't initially load content RESOLVED WONTFIX

26 Total; 4 Open (15.38%); 18 Resolved (69.23%); 4 Verified (15.38%);


Potential Explosive Concerns

Bug Mitigation
bug 860641 startup crash in ffi_call with RelevantKnowledge 1.0.0.2 Should be RelNoted to educate workaround
bug 767343 crash in nsSupportsStringImpl::SetData Year old crash trending up slightly, need to investigate further

Firefox 23.0.1

Build Info

Build 1 Build 2
ID 20130814063812
Builds FTP, HTTP
Changelog a55c55edf302
Configs ondemand.cfg

Automated Testing

Criteria Description Result Reports
Functional Automated functional tests via Mozmill-CI and ondemand PASS (Runtime: 66 min) mozmill-release (top-fails)
Betatest Automated software update tests on the betatest channel PASS (Runtime: 34 min) mozmill-release
Releasetest Automated software update tests on the releasetest channel
Release Automated software update tests on the release channel
Failures
Test Bug Count Status
testSecurity/testSafeBrowsingNotificationBar.js bug 905033 89 KNOWN TEST FAILURE (ASSIGNED)
testSearch/testSearchSuggestions.js bug 807251 10 KNOWN TEST FAILURE (REOPENED)
testAwesomeBar/testVisibleItemsMax.js bug 818128 10 KNOWN TEST FAILURE (ASSIGNED)

Manual Testing

WebRTC

Using the Moztrap test as a guideline, verify that Firefox 23.0.1 is no worse than Firefox 22.0 when making AppRTC calls of >5 minutes in length. When testing, here are the guidelines to follow:

  • Have only one browser open on each machine at any given time
  • Have only one call running at any given time on a particular machine
  • Make sure the caller and callee are always on *different* machines.
  • Please don't test any 3+-way calls for this sanity check -- We just want to see the results for 1:1 (basic) calling
  • If you find regressions, report a bug and CC Randell and Maire; they can help track down if the regressions are real or not
    • Tip: be sure to provide extremely detailed steps to reproduce and witnessed results, as well as detailed information about your test environment; more information is better than not enough information.
Caller Callee Result
Firefox 23.0.1 on Windows 7 Firefox 23.0.1 on Windows 7 [DONE]
Firefox 22.0 on Windows 7 Firefox 23.0.1 on Windows XP [DONE]
Chrome Stable on Windows 7 Firefox 23.0.1 on Windows 8 [DONE]
Firefox 23.0.1 on Windows 7 Firefox 22.0 on Mac OSX 10.8 [DONE]
Firefox 23.0.1 on Windows 7 Chrome Stable on Ubuntu 12.04 [DONE]
Firefox 22.0 on Windows XP Firefox 23.0.1 on Windows 7 [DONE]
Firefox 23.0.1 on Windows XP Firefox 22.0 on Windows XP [DONE]
Firefox 23.0.1 on Windows XP Chrome Stable on Windows 8 [DONE]
Chrome Stable on Windows XP Firefox 23.0.1 on Mac OSX 10.8 [ON TRACK]
Firefox 22.0 on Windows XP Firefox 23.0.1 on Ubuntu 13.04 [ON TRACK]
Chrome Stable on Windows 8 Firefox 23.0.1 on Windows 7 [DONE]
Firefox 23.0.1 on Windows 8 Chrome Stable on Windows XP [DONE]
Firefox 22.0 on Windows 8 Firefox 23.0.1 on Windows 8 [DONE]
Firefox 23.0.1 on Windows 8 Firefox 22.0 on Windows 8 [DONE]
Firefox 22.0 on Windows 8 Firefox 23.0.1 on Mac OSX 10.8 [DONE]
Chrome Stable on Windows 8 Firefox 23.0.1 on Ubuntu 13.04 [DONE]
Firefox 23.0.1 on Mac OSX 10.8 Firefox 22.0 on Windows 7 [DONE]
Chrome Stable on Mac OSX 10.8 Firefox 23.0.1 on Windows XP [DONE]
Firefox 22.0 on Mac OSX 10.8 Firefox 23.0.1 on Windows 8 [DONE]
Firefox 23.0.1 on Mac OSX 10.8 Chrome Stable on Mac OSX 10.8 [ON TRACK]
Firefox 23.0.1 on Mac OSX 10.8 Firefox 22.0 on Ubuntu 12.04 [ON TRACK]
Firefox 23.0.1 on Ubuntu 12.04 Chrome Stable on Windows 7 [DONE]
Firefox 22.0 on Ubuntu 12.04 Firefox 23.0.1 on Windows XP [DONE]
Chrome Stable on Ubuntu 12.04 Firefox 23.0.1 on Windows 8 [DONE]
Firefox 23.0.1 on Ubuntu 12.04 Firefox 22.0 on Mac OSX 10.8 [DONE]
Firefox 23.0.1 on Ubuntu 12.04 Chrome Stable on Ubuntu 12.04 [DONE]
Notes & Issues Found
  • no issues found

Basic Installation and Startup

Using the Moztrap test as a guideline, test that en-US and l10n builds install and function correctly.

Platform Locales Result
Windows XP en-US & de [ON TRACK]
Windows Vista en-US & fr [DONE]
Windows 7 en-US & it [DONE]
Windows 8 en-US & pt-BR [DONE]
Windows 8.1 en-US & zh-TW
Notes & Issues Found
  • For the stub installer part, only the installer and starting the browser was tested, since it installs Firefox 23.0, not 23.0.1.

Bug Fix Verifications

Bug Status
bug 901944 Rendering glitches on H.264 video only in FF23 on Vista VERIFIED FIXED by Samvedana Gohil
bug 902173 "[fr] Outils" wrongly translated to "Outils de développement" in menu since FF 23 VERIFIED FIXED by Ioana Budnar
bug 902532 Spellchecking broken with non-ASCII characters in profile path VERIFIED FIXED by Ioana Budnar
bug 901527 Audio static/"burble"/breakup in mozilla to mozilla webrtc calls
bug 902349 crash in nsStyleSet::FileRules with AMD Radeon HD 6310 RESOLVED FIXED: Not verifiable by QA
bug 904001 Block rlnx.dll, pmnx.dll, opnx.dll, prnx.dll 1.3.334.9 (Relevant Knowledge 1.0.2) RESOLVED FIXED: Not verifiable by QA

Post-Mortem

Successes

The following is a list of things which did go well with Firefox 23 from Nightly through to Release.

  • Developer Tanvi Vyas went above and beyond, working with QA (and many other parties) to make sure her feature (Mixed Content Blocking) was well-understood and well-tested.

Failures

The following is a list of things which did not go well with Firefox 23 from Nightly through to Release.

  • Missed to catch WebRTC bug 901527
  • Missed to catch h.264 bug 901944
    • What happened here proves a quite serious communication problem between QA and dev. The QA working on the H264... support feature was told it's only for Windows 7 and 8 and never got further updates. bug 847267 (which caused bug 901944) wasn't added as a dependency to the tracking bug either bug 799318. The feature information tracking should get improved. Otherwise we might get left with the only option of spamming devs with emails requesting for updates every week (a situation unpleasant for everyone).
  • Features lists for this release were not centralized anywhere, so at least one feature did not get to QA: Network Monitor.
  • The beta scope was not adapted to the new 2 betas per week process, which created QA overhead.
  • Invalid/unclear tests added to the regression test suite.
    • Unclear tests should be updated. If their owner can't do that, the QA team is willing to take on that.
    • Invalid tests should be removed from the suite as they seem to lead to invalid bugs. They could also be updated so they become valid.

Mitigating Strategy

The following is a list of actions we can/should take to build on our successes and prevent similar failures.

  • bug 901527 - WebRTC was not tested on beta (as visible in the above test plan). We should make sure our approach for following releases testing will include both the new and newly buggy areas.

Meeting Details

Retrieved from "https://wiki.allizom.org/index.php?title=Releases/Firefox_23/Test_Plan&oldid=695551"