Releases/Firefox 16/Test Plan: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
Line 2,665: Line 2,665:
  |-
  |-
  | Mac OS X 10.7
  | Mac OS X 10.7
  | tester
  | Virgil
  | '''RESULT''' - 2012-10-24
  | '''In progress''' - 2012-10-24
  |-
  |-
  | Mac OS X 10.8
  | Mac OS X 10.8

Revision as of 06:31, 25 October 2012

« Firefox 15 | Firefox 16 | Firefox 17 »

Firefox 16 Test Plan

Summary

Milestone Firefox 16
Status Currently in Beta -- merging to Release on October 8, 2012
Lead Juan Becerra (irc: juanb)
2nd Anthony Hughes (irc: ashughes)
Builds beta.mozilla.org

The following is the test plan for Firefox 16 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 16. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Schedule

The following is a description of the tasks which need to be completed throughout this release.

  • 2012-06-04: [DONE] Nightly Merge
  • 2012-07-17: [DONE] Aurora Merge
  • 2012-08-28: [DONE] Pre-Beta feature sign-offs
  • 2012-08-30: [DONE] Beta 1
  • 2012-09-06: [DONE] Beta 2
  • 2012-09-13: [ON TRACK] Beta 3
  • 2012-09-20: {{|}} Beta 4
  • 2012-09-27: {{|}} Beta 5
  • 2012-10-04: {{|}} Beta 6
  • 2012-10-06: {{|}} Release Candidate
  • 2012-10-09: {{|}} Final

Nightly

Merge

Functional Automation - result
Update Spotchecks - result

Features

See the Feature Release Tracking wiki for a general overview

Feature Owner Status Tests
Incremental GC Ioana Budnar On track
Opt-in activation for plugins Paul Silaghi On track
Developer Toolbar Virgil Dicu On track
Command Line Mihaela Velimiroviciu On track
Show PDF inline Mihaela Velimiroviciu Disabled until Fx18(bug 786012)
Panel-based Download Manager Simona Badau Delayed to Fx18
OS X 10.7 support Vlad Ghetiu Not implemented

Endurance Automation

The purpose of this section is to track any potential memory regressions via spikes reported in the dashboard.

What to Do
  1. Check the Endurance Charts
  2. If you see an anomalous event (ie. spike or dip), get the report for the specific event
  3. If the event happens twice in a row, file a bug under Mozmill Tests
  4. Notify Anthony Hughes of the issue
  5. Record the event in this wiki below
Current Status (Mihaela Velimiroviciu) - Last Updated Jul 18th, 2012
Full Query
ID Summary Priority Status
764823 Memory spike discovered by endurance tests on Nightly Linux and Mac between June 12 and 13 -- RESOLVED
767348 Memory consumption shows an ascending trend on Nightly/Win 6.1 x64 -- RESOLVED
773632 High results on mozmill endurance test on Mac 10.7.4_x64 since Jul 5 -- RESOLVED

3 Total; 0 Open (0%); 3 Resolved (100%); 0 Verified (0%);


Functional Automation

The purpose of this section is to track any potential functional regressions via results reported in the dashboard.

What to Do
  1. Check the testrun reports
  2. If you see any failing tests, check that there are not any bugs on file for it already
  3. If not, file a bug under Mozmill Tests
  4. Notify Anthony Hughes of the issue
  5. Record the event in this wiki below
Current Status (Vlad Maniac / Alex Lakatos) - Last Updated


Bug Triage

The purpose of this section is to triage UNCONFIRMED incoming bugs.

What to Do
  1. Check the UNCONFIRMED bugs
  2. Test the bugs in the latest Firefox Nightly
  3. If you can't reproduce the bug, ask for the reporter to provide more information
  4. If you reproduce the bug, add your testing information and mark the bug NEW
  5. Notify Anthony Hughes of the NEW issue
Current Status (Vlad Ghetiu / Virgil Dicu) - Last Updated

Result Spreadsheet


Aurora Merge

The following are the detailed results of the automated BFT test runs.

Platform Result: report
Windows 2000 PASS - 2012-07-19
Windows XP PASS - 2012-07-19
Windows Vista PASS - 2012-07-19
Windows 7 32-bit PASS - 2012-07-19
Windows 7 64-bit PASS - 2012-07-19
Mac OS X 10.6 PASS - 2012-07-19
Mac OS X 10.7 PASS - 2012-07-19
Ubuntu 11.04 32-bit PASS - 2012-07-19
Ubuntu 11.04 64-bit PASS - 2012-07-19
Known Issues
Full Query
ID Summary Priority Status
719982 Failure in testAddons_RestartlessExtensionWorksAfterRestart | Modal dialog has been found and processed P2 RESOLVED
770117 Test failure "Number of visible rows should equal max rows - '1' should equal '6'" in /testAwesomeBar/testVisibleItemsMax.js P2 RESOLVED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);


Updates

The following are details of update verifications for Firefox 15 merge to Aurora

Auroratest
  • {{|}} [tester] Windows XP:
  • {{|}} [tester] Windows 7:
  • {{|}} [tester] Mac OS X 10.6:
  • {{|}} [tester] Ubuntu 11.04:
Aurora
  • {{|}} [tester] Windows XP:
  • {{|}} [tester] Windows 7:
  • {{|}} [tester] Mac OS X 10.6:
  • {{|}} [tester] Ubuntu 11.04:
Issues Found

Manual Spot Checks and Feature Sign-Off

Are all features implemented at this point working?

Manual spot Checks [spreadsheet]

  • Platforms: Windows XP, Windows 7 x64, Ubuntu 11.10, Mac 10.6
  • Found known issues:
  • New issues:
  • Status:

Run the Firefox 16 Aurora Merge Feature Sign-Off tests in MozTrap and report results below.

Feature Owner Win32 Mac Linux
Panel Based Download Manager Simona Badau PASS - 2012-07-20 PASS - 2012-07-19 PASS - 2012-07-20
OS X 10.7 support - Not landed Vlad Ghetiu N/A - 2012-07-19 N/A - 2012-07-19 N/A - 2012-07-19
Opt-in activation for plugins Paul Silaghi PASS - 2012-07-20 PASS - 2012-07-20 PASS - 2012-07-20
Speedy Session Restore Paul Silaghi PASS - 2012-07-19 PASS - 2012-07-19 PASS - 2012-07-19
Developer Toolbar Virgil Dicu PASS - 2012-07-19 PASS - 2012-07-19 PASS - 2012-07-19
Command Line Mihaela Velimiroviciu PASS - 2012-07-20 PASS - 2012-07-20 PASS - 2012-07-20
Show PDF Inline Mihaela Velimiroviciu PASS - 2012-07-20 PASS - 2012-07-20 PASS - 2012-07-20
Incremental Garbage Collection Ioana Budnar PASS - 2012-07-19 PASS - 2012-07-20 PASS - 2012-07-20

Notes:

  • OS X 10.7 support - not landed

Smoke tests

Run the smoketests in MozTrap and report results below.

Platform Assignee Result: report
Windows XP Simona PASS - 2012-07-19
Windows 7 64-bit Paul PASS - 2012-07-20
Mac OS X 10.7.4 VladG PASS - 2012-07-19
Ubuntu 12.04 32-bit Virgil PASS - 2012-07-19

Known Issues

Full Query
ID Summary Priority Status
692446 Firefox prompts to install windows media player plugin even though it is already installed -- RESOLVED
775491 HTML5 unresponsive script errors on YouTube (unresponsive script name: html5player-vflGWJljH.js) -- RESOLVED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);


Pre-beta Feature testing and Sign-off

Manual spot Checks [spreadsheet]

  • Platforms: Windows XP, Windows 7 x64, Ubuntu 11.10, Mac 10.6
  • Found known issues:
  • New issues:
  • Status:

Run the Firefox 16 Aurora Merge Feature Sign-Off tests in MozTrap and report results below.

Feature Owner Win32 Mac Linux
In-content_preferences Mihaela Velimiroviciu disabled - 2012-08-24 disabled - 2012-08-24 disabled - 2012-08-24
Panel Based Download Manager Simona Badau disabled - 2012-08-24 disabled - 2012-08-24 disabled - 2012-08-24
OS X 10.7 support Vlad Ghetiu NA - 2012-08-24 NA - 2012-08-24 NA - 2012-08-24
Opt-in activation for plugins Paul Silaghi disabled - 2012-08-24 disabled - 2012-08-24 disabled - 2012-08-24
Speedy Session Restore Paul Silaghi NA - 2012-08-24 NA - 2012-08-24 NA - 2012-08-24
Developer Toolbar Virgil Dicu PASS - 2012-08-24 PASS - 2012-08-24 PASS - 2012-08-24
Command Line Mihaela Velimiroviciu PASS - 2012-08-24 PASS - 2012-08-24 PASS - 2012-08-24
Show PDF Inline Mihaela Velimiroviciu PASS - 2012-08-24 PASS - 2012-08-24 PASS - 2012-08-24
Incremental Garbage Collection Ioana Budnar PASS - 2012-08-24 PASS - 2012-08-24 PASS - 2012-08-24

Notes:

  • OS X 10.7 support: Not yet implemented.
  • Speedy Session Restore: backed out.

Smoke tests

Run the smoketests in MozTrap and report results below.

Platform Assignee Result: report
Windows XP Simona PASS - 2012-08-24
Windows 7 64-bit VladG PASS - 2012-08-24
Mac OS X 10.7.4 VladG PASS - 2012-08-24
Ubuntu 12.04 32-bit Virgil PASS - 2012-08-24

Known Issues

Full Query
ID Summary Priority Status
775082 Video controls disappear after dragging the tabs -- RESOLVED
784289 Inspector button remains open in devtoolbar when closing the Inspector via x button -- RESOLVED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);


Beta 1

Builds FTP, HTTP
ID 20120828083259
Changeset c4e1ca772267 (pushlog)
Checklist

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform PASS: report
Windows XP PASS - 2012-08-28
Windows Vista PASS - 2012-08-28
Windows 7 32-bit PASS - 2012-08-28
Windows 7 64-bit PASS - 2012-08-28
Windows 8 32-bit N/A (pending IT)
Windows 8 64-bit N/A (pending IT)
Mac OS X 10.6 PASS - 2012-08-28
Mac OS X 10.7 PASS - 2012-08-28
Mac OS X 10.8 PASS - 2012-08-28
Ubuntu 12.04 32-bit PASS - 2012-08-28
Ubuntu 12.04 64-bit PASS - 2012-08-28
Test Failures (known bugs)

Manual Testing

Operating System Tester Plug-in Compatibility Add-on Compatibility Web Compatibility
Windows XP Simona PASS - 2012-08-29 PASS - 2012-08-29 PASS - 2012-08-29
Windows 7 Mihaela PASS - 2012-08-29 PASS - 2012-08-29 PASS - 2012-08-29
Windows 8 Paul PASS - 2012-08-29 PASS - 2012-08-29 PASS - 2012-08-29
Mac OS X VladG PASS - 2012-08-29 PASS - 2012-08-29 PASS - 2012-08-29
Ubuntu 12.04 Virgil PASS - 2012-08-29 PASS - 2012-08-29 PASS - 2012-08-29
Operating System Tester Web Mail Mac 10.8 Signing Breakpad
Windows XP Simona PASS - 2012-08-29 N/A PASS - 2012-08-29
Windows 7 Mihaela PASS - 2012-08-29 N/A PASS - 2012-08-29
Windows 8 Paul PASS - 2012-08-29 N/A PASS - 2012-08-29
Mac OS X VladG PASS - 2012-08-29 PASS - 2012-08-29 PASS - 2012-08-29
Ubuntu 12.04 Virgil PASS - 2012-08-29 N/A PASS - 2012-08-29

Plug-in Compatibility

Install the following plug-ins and go to websites which use them, looking for rendering issues, functional issues, performance issues, and crashes.

Add-on Compatibility

Test the following websites for rendering issues, functional issues, performance issues, and crashes. Pay particular attention to any problems seen after updating Firefox with add-ons installed.

  • Top AMO add-ons
  • AVG toolbar (installed via AVG software)
    • AVG Safe Search 12.0.0.1912 is not compatible with Firefox 15 beta.
    • AVG Security Toolbar can't be installed on Win XP - we get a blue screen while installing it.
  • Norton toolbar (installed via Norton software)
    • Not currently available. Investigating if we can get this software.
  • Yahoo toolbar
  • Yandex toolbar
  • Babylon toolbar
  • Ad-Aware Antivirus
    • The toolbar can't be installed on Firefox.

Web Compatibility

Test the following websites for rendering issues, functional issues, performance issues, and crashes

  1. Overall Top 10
  2. Top by Country (pick a random site from three random countries)
  3. Top by Category (pick a random site from three random categories)
  4. Webmail clients like gmail, hotmail, and yahoo mail (pay particular attention to any built in web messenger software)
  5. Try Google's MapsGL demo
  6. Try YouTube's HTML5 demo

Web Mail

Test the following websites for rendering issues, functional issues, performance issues, and crashes. In particular make sure email, attachments, web messenger, and video chat are working as expected.

  1. Google Mail
  2. Yahoo Mail
  3. Hotmail
  4. Outlook.com

Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
  • Open a terminal window and cd to the folder Firefox.app is located
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues

Spot check build after this test

  • Try all the menu item functions
  • Gestures
  • Plug-in compatibility
  • Top-site compatibility
  • Add-on compatibility
  • Software update

Breakpad End-to-End

Test the end-to-end functionality in the crash reporting feature, across operating systems.

  • Use the Crash Me add-on to trigger a crash
  • Set up a dirty profile with lots of history, bookmarks, add-ons, themes, tabs, app-tabs, plugin content loaded, etc
  • Repeatedly crash Firefox and submit your reports
  • Verify that your session restores properly
  • Verify your crash reports are submitted properly by going through about:crashes and clicking the reports
  • Try adding comments to your crash report and verify the crash-stats.com report shows the comment

Update Testing

  • Partial: 15.0b6
  • Complete: 15.0b5 and earlier
Platform Releasetest: report Beta: report
Windows XP PASS - 2012-08-29 PASS - 2012-08-30
Windows Vista PASS - 2012-08-29 PASS - 2012-08-30
Windows 7 32-bit PASS - 2012-08-29 PASS - 2012-08-30
Windows 7 64-bit PASS - 2012-08-29 PASS - 2012-08-30
Windows 8 32-bit manual - see below manual - see below
Windows 8 64-bit manual - see below manual - see below
Mac OS X 10.6 PASS - 2012-08-29 PASS - 2012-08-30
Mac OS X 10.7 PASS - 2012-08-29 PASS - 2012-08-30
Mac OS X 10.8 PASS - 2012-08-29 PASS - 2012-08-30
Ubuntu 12.04 32-bit PASS - 2012-08-29 PASS - 2012-08-30
Ubuntu 12.04 64-bit PASS - 2012-08-29 PASS - 2012-08-30

Windows 8: manual spotcheck until automation in place

  • RELEASETEST
    • PASS - Firefox 15.0b6 en-US (partial)
    • PASS - Firefox 15.0b6 de (partial)
    • PASS - Firefox 14.0b6 en-US (complete)
    • PASS - Firefox 14.0b6 zh-TW (complete)
  • BETA
    • PASS - Firefox 15.0b6 en-US (partial)
    • PASS - Firefox 15.0b6 de (partial)
    • PASS - Firefox 14.0b6 en-US (complete)
    • PASS - Firefox 14.0b6 zh-TW (complete)
Issues

Bug Triage

  1. VERIFYME (3): verify the fix and do some exploratory testing around the issue
  2. FIXED (221): triage these bugs and test to ensure the fix is effective
  3. QAWANTED (3): triage these bugs and test to fulfill the QA request
  4. UNCONFIRMED (3): triage these bugs for potential new regressions
Results

Beta 2

Builds FTP, HTTP
ID 20120904124322
Changeset cad589192f3c (pushlog)
Checklist

Automated Testing 2

The following are the detailed results of the automated Mozmill test runs.

Platform PASS: report
Windows XP PASS - 2012-09-05
Windows Vista PASS - 2012-09-05
Windows 7 32-bit PASS - 2012-09-05
Windows 7 64-bit PASS - 2012-09-05
Windows 8 32-bit N/A (pending IT)
Windows 8 64-bit N/A (pending IT)
Mac OS X 10.6 PASS - 2012-09-05
Mac OS X 10.7 PASS - 2012-09-05
Mac OS X 10.8 PASS - 2012-09-05
Ubuntu 12.04 32-bit PASS - 2012-09-05
Ubuntu 12.04 64-bit PASS - 2012-09-05
Test Failures (known bugs)

Manual Testing 2

Operating System Tester Plug-in Compatibility Add-on Compatibility Web Compatibility
Windows XP Ioana PASS - 2012-09-07 PASS - 2012-09-07 PASS - 2012-09-07
Windows 7 Mihaela PASS - 2012-09-07 PASS - 2012-09-07 PASS - 2012-09-07
Windows 8 Virgil PASS - 2012-09-07 PASS - 2012-09-07 PASS - 2012-09-07
Mac OS X VladG PASS - 2012-09-07 PASS - 2012-09-07 PASS - 2012-09-07
Ubuntu 12.04 Ioana PASS - 2012-09-07 PASS - 2012-09-07 PASS - 2012-09-07
Operating System Tester Web Mail Mac 10.8 Signing Breakpad
Windows XP Virgil PASS - 2012-09-07 N/A PASS - 2012-09-07
Windows 7 Mihaela PASS - 2012-09-07 N/A PASS - 2012-09-07
Windows 8 Virgil PASS - 2012-09-07 N/A PASS - 2012-09-07
Mac OS X VladG PASS - 2012-09-07 PASS - 2012-09-07 PASS - 2012-09-07
Ubuntu 12.04 Ioana PASS - 2012-09-07 N/A PASS - 2012-09-07

Plug-in Compatibility 2

Install the following plug-ins and go to websites which use them, looking for rendering issues, functional issues, performance issues, and crashes.

Add-on Compatibility 2

Test the following websites for rendering issues, functional issues, performance issues, and crashes. Pay particular attention to any problems seen after updating Firefox with add-ons installed.

Web Compatibility 2

Test the following websites for rendering issues, functional issues, performance issues, and crashes

  1. Overall Top 10
  2. Top by Country (pick a random site from three random countries)
  3. Top by Category (pick a random site from three random categories)
  4. Webmail clients like gmail, hotmail, and yahoo mail (pay particular attention to any built in web messenger software)
  5. Try Google's MapsGL demo
  6. Try YouTube's HTML5 demo

Web Mail 2

Test the following websites for rendering issues, functional issues, performance issues, and crashes. In particular make sure email, attachments, web messenger, and video chat are working as expected.

  1. Google Mail
  2. Yahoo Mail
  3. Hotmail
  4. Outlook.com

Mac 10.8 Signing 2

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
  • Open a terminal window and cd to the folder Firefox.app is located
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues

Spot check build after this test

  • Try all the menu item functions
  • Gestures
  • Plug-in compatibility
  • Top-site compatibility
  • Add-on compatibility
  • Software update

Breakpad End-to-End 2

Test the end-to-end functionality in the crash reporting feature, across operating systems.

  • Use the Crash Me add-on to trigger a crash
  • Set up a dirty profile with lots of history, bookmarks, add-ons, themes, tabs, app-tabs, plugin content loaded, etc
  • Repeatedly crash Firefox and submit your reports
  • Verify that your session restores properly
  • Verify your crash reports are submitted properly by going through about:crashes and clicking the reports
  • Try adding comments to your crash report and verify the crash-stats.com report shows the comment

Update Testing 2

  • Partial: 16.0b1
  • Complete: 15.0b6 and earlier
Platform Releasetest: report Beta: report
Windows XP PASS - 2012-09-06 PASS - 2012-09-06
Windows Vista PASS - 2012-09-06 PASS - 2012-09-06
Windows 7 32-bit PASS - 2012-09-06 PASS - 2012-09-06
Windows 7 64-bit PASS - 2012-09-06 PASS - 2012-09-06
Windows 8 32-bit manual - see below manual - see below
Windows 8 64-bit PASS - see below PASS - see below
Mac OS X 10.6 PASS - 2012-09-06 PASS - 2012-09-06
Mac OS X 10.7 PASS - 2012-09-06 PASS - 2012-09-06
Mac OS X 10.8 PASS - 2012-09-06 PASS - 2012-09-06
Ubuntu 12.04 32-bit PASS - 2012-09-06 PASS - 2012-09-06
Ubuntu 12.04 64-bit PASS - 2012-09-06 PASS - 2012-09-06

Windows 8: manual spotcheck until automation in place

  • RELEASETEST
    • PASS - Firefox 16.0b1 en-US (partial)
    • PASS - Firefox 15.0b6 de (complete)
  • BETA
    • PASS - Firefox 16.0b1 en-US (partial)
    • PASS - Firefox 15.0b6 de (complete)
Issues

Bug Triage 2

  1. VERIFYME : verify the fix and do some exploratory testing around the issue
  2. FIXED : triage these bugs and test to ensure the fix is effective
  3. QAWANTED : triage these bugs and test to fulfill the QA request
  4. UNCONFIRMED : triage these bugs for potential new regressions
Results

Beta 3

Builds FTP, HTTP
ID 20120911232325
Changeset de3f06a549de (pushlog)
Checklist

Automated Testing 3

The following are the detailed results of the automated Mozmill test runs.

Platform PASS: report
Windows XP PASS - 2012-09-12
Windows Vista PASS - 2012-09-12
Windows 7 32-bit PASS - 2012-09-12
Windows 7 64-bit PASS - 2012-09-12
Windows 8 32-bit PASS - 2012-09-12
Windows 8 64-bit PASS - 2012-09-12
Mac OS X 10.6 PASS - 2012-09-12
Mac OS X 10.7 PASS - 2012-09-12
Mac OS X 10.8 PASS - 2012-09-12
Ubuntu 12.04 32-bit PASS - 2012-09-12
Ubuntu 12.04 64-bit PASS - 2012-09-12
Test Failures (known bugs)

Manual Testing 3

Operating System Tester Plug-in Compatibility Add-on Compatibility Web Compatibility
Windows XP Mihaela PASS - 2012-09-12 PASS - 2012-09-13 PASS - 2012-09-13
Windows 7 Ioana PASS - 2012-09-12 PASS - 2012-09-13 PASS - 2012-09-13
Windows 8 Paul PASS - 2012-09-12 PASS - 2012-09-12 PASS - 2012-09-12
Mac OS X VladG PASS - 2012-09-12 PASS - 2012-09-12 PASS - 2012-09-12
Ubuntu 12.04 Virgil PASS - 2012-09-12 PASS - 2012-09-12 PASS - 2012-09-12
Operating System Tester Web Mail Mac 10.8 Signing Breakpad
Windows XP Mihaela PASS - 2012-09-13 N/A PASS - 2012-09-13
Windows 7 Ioana PASS - 2012-09-13 N/A PASS - 2012-09-13
Windows 8 Paul PASS - 2012-09-13 N/A PASS - 2012-09-13
Mac OS X VladG PASS - 2012-09-12 PASS - 2012-09-12 PASS - 2012-09-12
Ubuntu 12.04 Virgil PASS - 2012-09-12 N/A PASS - 2012-09-12

Plug-in Compatibility 3

Install the following plug-ins and go to websites which use them, looking for rendering issues, functional issues, performance issues, and crashes.

Add-on Compatibility 3

Test the following websites for rendering issues, functional issues, performance issues, and crashes. Pay particular attention to any problems seen after updating Firefox with add-ons installed.

Web Compatibility 3

Test the following websites for rendering issues, functional issues, performance issues, and crashes

  1. Overall Top 10
  2. Top by Country (pick a random site from three random countries)
  3. Top by Category (pick a random site from three random categories)
  4. Webmail clients like gmail, hotmail, and yahoo mail (pay particular attention to any built in web messenger software)
  5. Try Google's MapsGL demo
  6. Try YouTube's HTML5 demo

Web Mail 3

Test the following websites for rendering issues, functional issues, performance issues, and crashes. In particular make sure email, attachments, web messenger, and video chat are working as expected.

  1. Google Mail
  2. Yahoo Mail
  3. Hotmail
  4. Outlook.com

Mac 10.8 Signing 3

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
  • Open a terminal window and cd to the folder Firefox.app is located
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues

Spot check build after this test

  • Try all the menu item functions
  • Gestures
  • Plug-in compatibility
  • Top-site compatibility
  • Add-on compatibility
  • Software update

Breakpad End-to-End 3

Test the end-to-end functionality in the crash reporting feature, across operating systems.

  • Use the Crash Me add-on to trigger a crash
  • Set up a dirty profile with lots of history, bookmarks, add-ons, themes, tabs, app-tabs, plugin content loaded, etc
  • Repeatedly crash Firefox and submit your reports
  • Verify that your session restores properly
  • Verify your crash reports are submitted properly by going through about:crashes and clicking the reports
  • Try adding comments to your crash report and verify the crash-stats.com report shows the comment

Update Testing 3

  • Partial: 16.0b2
  • Complete: 15.0b6 and earlier
Platform Releasetest: report Beta: report
Windows XP PASS - 2012-09-13 PASS - 2012-09-13
Windows Vista PASS - 2012-09-13 PASS - 2012-09-13
Windows 7 32-bit PASS - 2012-09-13 PASS - 2012-09-13
Windows 7 64-bit PASS - 2012-09-13 PASS - 2012-09-13
Windows 8 32-bit manual-see below - 2012-09-13 Results - yyyy-mm-dd
Windows 8 64-bit manual-see below - 2012-09-13 Results - yyyy-mm-dd
Mac OS X 10.6 PASS - 2012-09-13 PASS - 2012-09-13
Mac OS X 10.7 PASS - 2012-09-13 PASS - 2012-09-13
Mac OS X 10.8 PASS - 2012-09-13 PASS - 2012-09-13
Ubuntu 12.04 32-bit PASS - 2012-09-13 PASS - 2012-09-13
Ubuntu 12.04 64-bit PASS - 2012-09-13 PASS - 2012-09-13

Windows 8: manual spotcheck until automation in place

  • RELEASETEST
    • PASS - Firefox 16.0b1 en-US (partial)
    • PASS - Firefox 15.0b6 de (complete)
  • BETA
    • PASS - Firefox 16.0b2 en-US (partial)
    • PASS - Firefox 15.0b6 de (complete)
Issues

Bug Triage 3

  1. VERIFYME (3) : verify the fix and do some exploratory testing around the issue
  2. FIXED (184) : triage these bugs and test to ensure the fix is effective
  3. QAWANTED (8) : triage these bugs and test to fulfill the QA request
  4. UNCONFIRMED (10) : triage these bugs for potential new regressions
Results

Beta 4

Builds FTP, HTTP
ID 20120919065210
Changeset c3be659f6121
Checklist

Automated Testing 4

The following are the detailed results of the automated Mozmill test runs.

Platform RESULT: report
Windows XP PASS - 2012-09-19
Windows Vista PASS - 2012-09-19
Windows 7 32-bit PASS - 2012-09-19
Windows 7 64-bit PASS - 2012-09-19
Windows 8 32-bit PASS - 2012-09-19
Windows 8 64-bit PASS - 2012-09-19
Mac OS X 10.6 PASS - 2012-09-19
Mac OS X 10.7 PASS - 2012-09-19
Mac OS X 10.8 PASS - 2012-09-19
Ubuntu 12.04 32-bit PASS - 2012-09-19
Ubuntu 12.04 64-bit PASS - 2012-09-19
Test Failures (known bugs)

Manual Testing 4

Operating System Tester Plug-in Compatibility Add-on Compatibility Web Compatibility
Windows XP Vlad, Mihaela PASS - 2012-09-21 PASS - 2012-09-21 PASS - 2012-09-21
Windows 7 Simona PASS - 2012-09-20 PASS - 2012-09-20 PASS - 2012-09-20
Windows 8 Paul PASS - 2012-09-20 PASS - 2012-09-20 PASS - 2012-09-20
Mac OS X VladG PASS - 2012-09-20 PASS - 2012-09-20 PASS - 2012-09-20
Ubuntu 12.04 Mihaela PASS - 2012-09-20 PASS - 2012-09-20 PASS - 2012-09-20
Operating System Tester Web Mail Mac 10.8 Signing Breakpad
Windows XP Mihaela PASS - 2012-09-20 N/A PASS - 2012-09-20
Windows 7 Simona PASS - 2012-09-20 N/A PASS - 2012-09-20
Windows 8 Paul PASS - 2012-09-20 N/A PASS - 2012-09-20
Mac OS X VladG PASS - 2012-09-20 PASS - 2012-09-20 PASS - 2012-09-20
Ubuntu 12.04 Mihaela PASS - 2012-09-20 N/A PASS - 2012-09-20

Plug-in Compatibility 4

Install the following plug-ins and go to websites which use them, looking for rendering issues, functional issues, performance issues, and crashes.

Add-on Compatibility 4

Test the following websites for rendering issues, functional issues, performance issues, and crashes. Pay particular attention to any problems seen after updating Firefox with add-ons installed.

Web Compatibility 4

Test the following websites for rendering issues, functional issues, performance issues, and crashes

  1. Overall Top 10
  2. Top by Country (pick a random site from three random countries)
  3. Top by Category (pick a random site from three random categories)
  4. Webmail clients like gmail, hotmail, and yahoo mail (pay particular attention to any built in web messenger software)
  5. Try Google's MapsGL demo
  6. Try YouTube's HTML5 demo

Web Mail 4

Test the following websites for rendering issues, functional issues, performance issues, and crashes. In particular make sure email, attachments, web messenger, and video chat are working as expected.

  1. Google Mail
  2. Yahoo Mail
  3. Hotmail
  4. Outlook.com

Mac 10.8 Signing 4

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
  • Open a terminal window and cd to the folder Firefox.app is located
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues

Spot check build after this test

  • Try all the menu item functions
  • Gestures
  • Plug-in compatibility
  • Top-site compatibility
  • Add-on compatibility
  • Software update

Breakpad End-to-End 4

Test the end-to-end functionality in the crash reporting feature, across operating systems.

  • Use the Crash Me add-on to trigger a crash
  • Set up a dirty profile with lots of history, bookmarks, add-ons, themes, tabs, app-tabs, plugin content loaded, etc
  • Repeatedly crash Firefox and submit your reports
  • Verify that your session restores properly
  • Verify your crash reports are submitted properly by going through about:crashes and clicking the reports
  • Try adding comments to your crash report and verify the crash-stats.com report shows the comment

Update Testing 4

  • Partial: 16.0b3
  • Complete: 16.0b2 and earlier
Platform Releasetest: report Beta: report
Windows XP PASS - 2012-09-20 RESULT - 2012-09-19
Windows Vista PASS - 2012-09-20 RESULT - 2012-09-19
Windows 7 32-bit PASS - 2012-09-20 RESULT - 2012-09-19
Windows 7 64-bit PASS - 2012-09-20 RESULT - 2012-09-19
Windows 8 32-bit manual-see below - 2012-09-19 Results - yyyy-mm-dd
Windows 8 64-bit manual-see below - 2012-09-19 Results - yyyy-mm-dd
Mac OS X 10.6 PASS - 2012-09-20 RESULT - 2012-09-19
Mac OS X 10.7 PASS - 2012-09-20 RESULT - 2012-09-19
Mac OS X 10.8 PASS - 2012-09-20 RESULT - 2012-09-19
Ubuntu 12.04 32-bit PASS - 2012-09-20 RESULT - 2012-09-19
Ubuntu 12.04 64-bit PASS - 2012-09-20 RESULT - 2012-09-19

Windows 8: manual spotcheck until automation in place

  • RELEASETEST
    • PASS - Firefox 16.0b3 en-US (partial)
    • PASS - Firefox 15.0b6 de (complete)
  • BETA
    • RESULT - Firefox 16.0b3 en-US (partial)
    • RESULT - Firefox 15.0b6 de (complete)
Issues

Bug Triage 4

  1. VERIFYME (?) : verify the fix and do some exploratory testing around the issue
  2. FIXED (?) : triage these bugs and test to ensure the fix is effective
  3. QAWANTED (?) : triage these bugs and test to fulfill the QA request
  4. UNCONFIRMED (?) : triage these bugs for potential new regressions
Results

Verification Spreadsheet


Beta 5

Builds FTP, HTTP
ID 20120925201946
Changeset 7c0af9b7ed61 (pushlog)
Checklist

Automated Testing 5

The following are the detailed results of the automated Mozmill test runs.

Platform RESULT: report
Windows XP PASS - 2012-09-26
Windows Vista PASS - 2012-09-26
Windows 7 32-bit PASS - 2012-09-26
Windows 7 64-bit PASS - 2012-09-26
Windows 8 32-bit PASS - 2012-09-26
Windows 8 64-bit PASS - 2012-09-26
Mac OS X 10.6 PASS - 2012-09-26
Mac OS X 10.7 PASS - 2012-09-26
Mac OS X 10.8 PASS - 2012-09-26
Ubuntu 12.04 32-bit PASS - 2012-09-26
Ubuntu 12.04 64-bit PASS - 2012-09-26
Test Failures (known bugs)

Manual Testing 5

Operating System Tester Plug-in Compatibility Add-on Compatibility Web Compatibility
Windows XP Simona PASS - 2012-09-26 PASS - 2012-09-26 PASS - 2012-09-26
Windows 7 Paul PASS - 2012-09-26 PASS - 2012-09-26 PASS - 2012-09-26
Windows 8 Mihaela PASS - 2012-09-26 PASS - 2012-09-26 PASS - 2012-09-26
Mac OS X VladG PASS - 2012-09-26 PASS - 2012-09-26 PASS - 2012-09-26
Ubuntu 12.04 Ioana PASS - 2012-09-26 PASS - 2012-09-26 PASS - 2012-09-27
Operating System Tester Web Mail Mac 10.8 Signing Breakpad
Windows XP Simona PASS - 2012-09-26 N/A PASS - 2012-09-26
Windows 7 Paul PASS - 2012-09-26 N/A PASS - 2012-09-26
Windows 8 Mihaela PASS - 2012-09-26 N/A PASS - 2012-09-27
Mac OS X VladG PASS - 2012-09-27 PASS - 2012-09-26 PASS - 2012-09-27
Ubuntu 12.04 Ioana PASS - 2012-09-27 N/A PASS - 2012-09-27

Plug-in Compatibility 5

Install the following plug-ins and go to websites which use them, looking for rendering issues, functional issues, performance issues, and crashes.

Add-on Compatibility 5

Test the following websites for rendering issues, functional issues, performance issues, and crashes. Pay particular attention to any problems seen after updating Firefox with add-ons installed.

Web Compatibility 5

Test the following websites for rendering issues, functional issues, performance issues, and crashes

  1. Overall Top 10
  2. Top by Country (pick a random site from three random countries)
  3. Top by Category (pick a random site from three random categories)
  4. Webmail clients like gmail, hotmail, and yahoo mail (pay particular attention to any built in web messenger software)
  5. Try Google's MapsGL demo
  6. Try YouTube's HTML5 demo

Web Mail 5

Test the following websites for rendering issues, functional issues, performance issues, and crashes. In particular make sure email, attachments, web messenger, and video chat are working as expected.

  1. Google Mail
  2. Yahoo Mail
  3. Hotmail
  4. Outlook.com

Mac 10.8 Signing 5

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
  • Open a terminal window and cd to the folder Firefox.app is located
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues

Spot check build after this test

  • Try all the menu item functions
  • Gestures
  • Plug-in compatibility
  • Top-site compatibility
  • Add-on compatibility
  • Software update

Breakpad End-to-End 5

Test the end-to-end functionality in the crash reporting feature, across operating systems.

  • Use the Crash Me add-on to trigger a crash
  • Set up a dirty profile with lots of history, bookmarks, add-ons, themes, tabs, app-tabs, plugin content loaded, etc
  • Repeatedly crash Firefox and submit your reports
  • Verify that your session restores properly
  • Verify your crash reports are submitted properly by going through about:crashes and clicking the reports
  • Try adding comments to your crash report and verify the crash-stats.com report shows the comment

Windows 8

Test Firefox integration with Windows 8 - etherpad.

Update Testing 5

  • Partial: 16.0b4
  • Complete: 16.0b3 and earlier
Platform Releasetest: report Beta: report
Windows XP PASS - 2012-09-26 RESULT - 2012-09-26
Windows Vista PASS - 2012-09-26 RESULT - 2012-09-26
Windows 7 32-bit PASS - 2012-09-26 RESULT - 2012-09-26
Windows 7 64-bit PASS - 2012-09-26 RESULT - 2012-09-26
Windows 8 32-bit manual-see below - 2012-09-19 Results - yyyy-mm-dd
Windows 8 64-bit manual-see below - 2012-09-19 Results - yyyy-mm-dd
Mac OS X 10.6 PASS - 2012-09-26 RESULT - 2012-09-26
Mac OS X 10.7 PASS - 2012-09-26 RESULT - 2012-09-26
Mac OS X 10.8 PASS - 2012-09-26 RESULT - 2012-09-26
Ubuntu 12.04 32-bit PASS - 2012-09-26 RESULT - 2012-09-26
Ubuntu 12.04 64-bit PASS - 2012-09-26 RESULT - 2012-09-26

Windows 8: manual spotcheck until automation in place

  • RELEASETEST
    • RESULT - Firefox 16.0b4 en-US (partial)
    • RESULT - Firefox 15.0b6 de (complete)
  • BETA
    • RESULT - Firefox 16.0b4 en-US (partial)
    • RESULT - Firefox 15.0b6 de (complete)
Issues

Bug Triage 5

  1. VERIFYME (19) : verify the fix and do some exploratory testing around the issue
  2. FIXED (184) : triage these bugs and test to ensure the fix is effective
  3. QAWANTED (6) : triage these bugs and test to fulfill the QA request
  4. UNCONFIRMED (14) : triage these bugs for potential new regressions
Results

Beta 6

Builds FTP, HTTP
ID 20121002073616
Changeset 8a8a932fbef4 (pushlog)
Checklist

Automated Testing 6

The following are the detailed results of the automated Mozmill test runs.

Platform RESULT: report
Windows XP PASS - 2012-10-02
Windows Vista PASS - 2012-10-02
Windows 7 32-bit PASS - 2012-10-02
Windows 7 64-bit PASS - 2012-10-02
Windows 8 32-bit PASS - 2012-10-02
Windows 8 64-bit PASS - 2012-10-02
Mac OS X 10.6 PASS - 2012-10-02
Mac OS X 10.7 PASS - 2012-10-02
Mac OS X 10.8 PASS - 2012-10-02
Ubuntu 12.04 32-bit PASS - 2012-10-02
Ubuntu 12.04 64-bit PASS - 2012-10-02
Test Failures (known bugs)

Manual Testing 6

Operating System Tester Plug-in Compatibility Add-on Compatibility Web Compatibility
Windows XP Paul/Virgil PASS - 2012-10-02 PASS - 2012-10-02 PASS - 2012-10-02
Windows 7 Ioana PASS - 2012-10-02 PASS - 2012-10-02 PASS - 2012-10-02
Windows 8 Simona PASS - 2012-10-02 PASS - 2012-10-02 PASS - 2012-10-02
Mac OS X VladG PASS - 2012-10-03 PASS - 2012-10-03 PASS - 2012-10-03
Ubuntu 12.04 Mihaela PASS - 2012-10-02 PASS - 2012-10-02 PASS - 2012-10-02
Operating System Tester Web Mail Mac 10.8 Signing Breakpad
Windows XP Paul/Virgil PASS - 2012-10-02 N/A PASS - 2012-10-02
Windows 7 Ioana/Paul PASS - 2012-10-02 N/A PASS - 2012-10-02
Windows 8 Simona PASS - 2012-10-02 N/A PASS - 2012-10-02
Mac OS X VladG PASS - 2012-10-03 PASS - 2012-10-03 PASS - 2012-10-03
Ubuntu 12.04 Mihaela PASS - 2012-10-02 N/A PASS - 2012-10-02

Plug-in Compatibility 6

Install the following plug-ins and go to websites which use them, looking for rendering issues, functional issues, performance issues, and crashes.

Add-on Compatibility 6

Test the following websites for rendering issues, functional issues, performance issues, and crashes. Pay particular attention to any problems seen after updating Firefox with add-ons installed.

Web Compatibility 6

Test the following websites for rendering issues, functional issues, performance issues, and crashes

  1. Overall Top 10
  2. Top by Country (pick a random site from three random countries)
  3. Top by Category (pick a random site from three random categories)
  4. Webmail clients like gmail, hotmail, and yahoo mail (pay particular attention to any built in web messenger software)
  5. Try Google's MapsGL demo
  6. Try YouTube's HTML5 demo

Web Mail 6

Test the following websites for rendering issues, functional issues, performance issues, and crashes. In particular make sure email, attachments, web messenger, and video chat are working as expected.

  1. Google Mail
  2. Yahoo Mail
  3. Hotmail
  4. Outlook.com

Mac 10.8 Signing 6

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
  • Open a terminal window and cd to the folder Firefox.app is located
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues

Spot check build after this test

  • Try all the menu item functions
  • Gestures
  • Plug-in compatibility
  • Top-site compatibility
  • Add-on compatibility
  • Software update

Breakpad End-to-End 6

Test the end-to-end functionality in the crash reporting feature, across operating systems.

  • Use the Crash Me add-on to trigger a crash
  • Set up a dirty profile with lots of history, bookmarks, add-ons, themes, tabs, app-tabs, plugin content loaded, etc
  • Repeatedly crash Firefox and submit your reports
  • Verify that your session restores properly
  • Verify your crash reports are submitted properly by going through about:crashes and clicking the reports
  • Try adding comments to your crash report and verify the crash-stats.com report shows the comment

Update Testing 6

  • Partial: 16.0b5
  • Complete: 16.0b4 and earlier
Platform Releasetest: report Beta: report
Windows XP PASS - 2012-10-02 RESULT - 2012-10-02
Windows Vista PASS - 2012-10-02 RESULT - 2012-10-02
Windows 7 32-bit PASS - 2012-10-02 RESULT - 2012-10-02
Windows 7 64-bit PASS - 2012-10-02 RESULT - 2012-10-02
Windows 8 32-bit manual-see below - 2012-10-02 Results - yyyy-mm-dd
Windows 8 64-bit manual-see below - 2012-10-02 Results - yyyy-mm-dd
Mac OS X 10.6 PASS - 2012-10-02 RESULT - 2012-10-02
Mac OS X 10.7 PASS - 2012-10-02 RESULT - 2012-10-02
Mac OS X 10.8 PASS - 2012-10-02 RESULT - 2012-10-02
Ubuntu 12.04 32-bit PASS - 2012-10-02 RESULT - 2012-10-02
Ubuntu 12.04 64-bit PASS - 2012-10-02 RESULT - 2012-10-02

Windows 8: manual spotcheck until automation in place

  • RELEASETEST
    • RESULT - Firefox 16.0b5 en-US (partial)
    • RESULT - Firefox 15.0b6 de (complete)
  • BETA
    • RESULT - Firefox 16.0b5 en-US (partial)
    • RESULT - Firefox 15.0b6 de (complete)
Issues

Bug Triage 6

  1. VERIFYME (10) : verify the fix and do some exploratory testing around the issue
  2. FIXED (181) : triage these bugs and test to ensure the fix is effective
  3. QAWANTED (4) : triage these bugs and test to fulfill the QA request
  4. UNCONFIRMED (18) : triage these bugs for potential new regressions
Results

Release Candidate

Builds FTP, HTTP
ID 20121005155445
Changeset 10fe550fadc6
Checklist

Automated Testing RC

The following are the detailed results of the automated Mozmill test runs.

Platform RESULT: report
Windows XP PASS - 2012-10-08
Windows Vista PASS - 2012-10-08
Windows 7 32-bit PASS - 2012-10-08
Windows 7 64-bit PASS - 2012-10-08
Windows 8 32-bit PASS - 2012-10-08
Windows 8 64-bit PASS - 2012-10-08
Mac OS X 10.6 PASS - 2012-10-08
Mac OS X 10.7 PASS - 2012-10-08
Mac OS X 10.8 PASS - 2012-10-08
Ubuntu 12.04 32-bit PASS - 2012-10-08
Ubuntu 12.04 64-bit PASS - 2012-10-08
Test Failures (known bugs)

Manual Testing RC

Operating System Tester Plug-in Compatibility Add-on Compatibility Web Compatibility
Windows XP Ioana PASS - 2012-10-09 PASS - 2012-10-09 PASS - 2012-10-09
Windows 7 Simona PASS - 2012-10-08 PASS - 2012-10-09 PASS - 2012-10-09
Windows 8 Mihaela PASS - 2012-10-09 PASS - 2012-10-09 PASS - 2012-10-09
Mac OS X VladG PASS - 2012-10-09 PASS - 2012-10-09 PASS - 2012-10-09
Ubuntu 12.04 Virgil PASS - 2012-10-08 PASS - 2012-10-09 PASS - 2012-10-08
Operating System Tester Web Mail Mac 10.8 Signing Breakpad
Windows XP Paul PASS - 2012-10-09 N/A PASS - 2012-10-09
Windows 7 Simona PASS - 2012-10-09 N/A PASS - 2012-10-09
Windows 8 Mihaela PASS - 2012-10-09 N/A PASS - 2012-10-09
Mac OS X VladG PASS - 2012-10-09 PASS - 2012-10-08 PASS - 2012-10-09
Ubuntu 12.04 Virgil PASS - 2012-10-08 N/A PASS - 2012-10-09

Plug-in Compatibility

Install the following plug-ins and go to websites which use them, looking for rendering issues, functional issues, performance issues, and crashes.

Add-on Compatibility

Test the following websites for rendering issues, functional issues, performance issues, and crashes. Pay particular attention to any problems seen after updating Firefox with add-ons installed.

Web Compatibility

Test the following websites for rendering issues, functional issues, performance issues, and crashes

  1. Overall Top 10
  2. Top by Country (pick a random site from three random countries)
  3. Top by Category (pick a random site from three random categories)
  4. Webmail clients like gmail, hotmail, and yahoo mail (pay particular attention to any built in web messenger software)
  5. Try Google's MapsGL demo
  6. Try YouTube's HTML5 demo

Web Mail

Test the following websites for rendering issues, functional issues, performance issues, and crashes. In particular make sure email, attachments, web messenger, and video chat are working as expected.

  1. Google Mail
  2. Yahoo Mail
  3. Hotmail
  4. Outlook.com

Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
  • Open a terminal window and cd to the folder Firefox.app is located
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues

Spot check build after this test

  • Try all the menu item functions
  • Gestures
  • Plug-in compatibility
  • Top-site compatibility
  • Add-on compatibility
  • Software update

Breakpad End-to-End

Test the end-to-end functionality in the crash reporting feature, across operating systems.

  • Use the Crash Me add-on to trigger a crash
  • Set up a dirty profile with lots of history, bookmarks, add-ons, themes, tabs, app-tabs, plugin content loaded, etc
  • Repeatedly crash Firefox and submit your reports
  • Verify that your session restores properly
  • Verify your crash reports are submitted properly by going through about:crashes and clicking the reports
  • Try adding comments to your crash report and verify the crash-stats.com report shows the comment

Update Testing RC

  • Partial: 15.0.1
  • Complete: 12.0 and earlier
Platform Releasetest: report Beta: report
Windows XP PASS - 2010-10-08 RESULTS - yyyy-mm-dd
Windows Vista PASS - 2010-10-08 RESULTS - yyyy-mm-dd
Windows 7 32-bit PASS - 2010-10-08 RESULTS - yyyy-mm-dd
Windows 7 64-bit PASS - 2010-10-08 RESULTS - yyyy-mm-dd
Windows 8 32-bit PASS- see below - 2012-10-09 RESULTS - yyyy-mm-dd
Windows 8 64-bit PASS- see below - 2012-10-08 RESULTS - yyyy-mm-dd
Mac OS X 10.6 PASS - 2010-10-08 RESULTS - yyyy-mm-dd
Mac OS X 10.7 PASS - 2010-10-08 RESULTS - yyyy-mm-dd
Mac OS X 10.8 PASS - 2010-10-08 RESULTS - yyyy-mm-dd
Ubuntu 12.04 32-bit PASS - 2010-10-08 RESULTS - yyyy-mm-dd
Ubuntu 12.04 64-bit PASS - 2010-10-08 RESULTS - yyyy-mm-dd

Windows 8: manual spotcheck until automation in place

  • BETATEST
    • PASS - Firefox 15.0.1 en-US de (partial)
    • PASS - Firefox 12.0 en-US fr (complete)
  • RELEASE
    • RESULT - Firefox 15.0.1 en-US de (partial)
    • RESULT - Firefox 12.0 en-US de (complete)
Issues

Bug Triage RC

  1. VERIFYME (9) : verify the fix and do some exploratory testing around the issue
  2. FIXED (178) : triage these bugs and test to ensure the fix is effective
  3. QAWANTED (8) : triage these bugs and test to fulfill the QA request
  4. UNCONFIRMED (21) : triage these bugs for potential new regressions
Results

16.0.1

Builds FTP, HTTP
ID 20121010144125
Changeset 7789ab68a99d (pushlog)
Checklist

Automated Testing 7

The following are the detailed results of the automated Mozmill test runs.

Platform RESULT: report
Windows XP PASS - 2012-10-10
Windows Vista PASS - 2012-10-10
Windows 7 32-bit PASS - 2012-10-10
Windows 7 64-bit PASS - 2012-10-10
Windows 8 32-bit PASS - 2012-10-10
Windows 8 64-bit PASS - 2012-10-10
Mac OS X 10.6 PASS - 2012-10-10
Mac OS X 10.7 PASS - 2012-10-10
Mac OS X 10.8 PASS - 2012-10-10
Ubuntu 12.04 32-bit PASS - 2012-10-10
Ubuntu 12.04 64-bit PASS - 2012-10-10
Test Failures (known bugs)

Manual Testing 7

There will be four bug verifications and testing around these bugs

Results
Platform Tester 16.0.1 Candidates
Windows XP juanb, ashughes, mwobensmith PASS - 2012-10-10
Windows Vista tester RESULT - 2012-10-10
Windows 7 32-bit tester RESULT - 2012-10-10
Windows 7 64-bit juanb, ashughes, mwobensmith PASS - 2012-10-10
Mac OS X 10.6 tester RESULT - 2012-10-10
Mac OS X 10.7 juanb, ashughes, mwobensmith PASS - 2012-10-10
Mac OS X 10.8 tester RESULT - 2012-10-10
Ubuntu 12.04 32-bit tester RESULT - 2012-10-10
Ubuntu 12.04 64-bit juanb, ashughes, mwobensmith PASS - 2012-10-10
Issues Found

Breakpad End-to-End 7

Test the end-to-end functionality in the crash reporting feature, across operating systems.

  • Use the Crash Me add-on to trigger a crash
  • Set up a dirty profile with lots of history, bookmarks, add-ons, themes, tabs, app-tabs, plugin content loaded, etc
  • Repeatedly crash Firefox and submit your reports
  • Verify that your session restores properly
  • Verify your crash reports are submitted properly by going through about:crashes and clicking the reports
  • Try adding comments to your crash report and verify the crash-stats.com report shows the comment

Update Testing 7

  • Partial: 13.0.1, 14.0.1, 15.0.1
  • Complete: 12.0, 13.0
Platform Releasetest: report Release: report
Windows XP PASS - 2012-10-11 PASS - 2012-10-11
Windows Vista PASS - 2012-10-11 PASS - 2012-10-11
Windows 7 32-bit PASS - 2012-10-11 PASS - 2012-10-11
Windows 7 64-bit PASS - 2012-10-11 PASS - 2012-10-11
Windows 8 32-bit Pass- juanb - 2012-10-11 Pass - juanb - 2012-10-11
Windows 8 64-bit Pass- juanb - 2012-10-11 Pass - 2012-10-11
Mac OS X 10.6 PASS - 2012-10-11 PASS - 2012-10-11
Mac OS X 10.7 PASS - 2012-10-11 PASS - 2012-10-11
Mac OS X 10.8 PASS - 2012-10-11 PASS - 2012-10-11
Ubuntu 12.04 32-bit PASS - 2012-10-11 PASS - 2012-10-11
Ubuntu 12.04 64-bit PASS - 2012-10-11 PASS - 2012-10-11

Windows 8: manual spotcheck until automation in place

  • RELEASETEST
    • PASS - Firefox 16.0 en-US (partial)
    • PASS - Firefox 15.0.1 de (partial)
  • RELEASE
    • PASS - Firefox 16.0 en-US (partial)
    • PASS - Firefox 12.0 fr (complete)
Issues

16.0.2

Builds FTP, HTTP
ID 20121024073032
Changeset 3995ff235342 (pushlog)
Checklist

Automated Testing 8

The following are the detailed results of the automated Mozmill test runs.

Platform RESULT: report
Windows XP PASS - 2012-10-24
Windows Vista PASS - 2012-10-24
Windows 7 32-bit PASS - 2012-10-24
Windows 7 64-bit PASS - 2012-10-24
Windows 8 32-bit PASS - 2012-10-24
Windows 8 64-bit PASS - 2012-10-24
Mac OS X 10.6 PASS - 2012-10-24
Mac OS X 10.7 PASS - 2012-10-24
Mac OS X 10.8 PASS - 2012-10-24
Ubuntu 12.04 32-bit PASS - 2012-10-24
Ubuntu 12.04 64-bit PASS - 2012-10-24
Test Failures (known bugs)

Manual Testing 8

There will be X bug verifications and testing around these bugs

Results
Platform Tester 16.0.2 Candidates
Windows XP tester RESULT - 2012-10-24
Windows Vista tester RESULT - 2012-10-24
Windows 7 32-bit tester RESULT - 2012-10-24
Windows 7 64-bit tester RESULT - 2012-10-24
Mac OS X 10.6 tester RESULT - 2012-10-24
Mac OS X 10.7 Virgil In progress - 2012-10-24
Mac OS X 10.8 tester RESULT - 2012-10-24
Ubuntu 12.04 32-bit Ioana In progress - 2012-10-25
Ubuntu 12.04 64-bit Ioana In progress - 2012-10-25
Issues Found

Breakpad End-to-End 8

Test the end-to-end functionality in the crash reporting feature, across operating systems.

  • Use the Crash Me add-on to trigger a crash
  • Set up a dirty profile with lots of history, bookmarks, add-ons, themes, tabs, app-tabs, plugin content loaded, etc
  • Repeatedly crash Firefox and submit your reports
  • Verify that your session restores properly
  • Verify your crash reports are submitted properly by going through about:crashes and clicking the reports
  • Try adding comments to your crash report and verify the crash-stats.com report shows the comment

Update Testing 8

  • Partial: 14.0.1, 15.0.1, 16.0.1
  • Complete: 12.0, 13.0
Platform Releasetest: report Beta: report
Windows XP RESULT - 2012-10-24 RESULT - 2012-10-24
Windows Vista RESULT - 2012-10-24 RESULT - 2012-10-24
Windows 7 32-bit RESULT - 2012-10-24 RESULT - 2012-10-24
Windows 7 64-bit RESULT - 2012-10-24 RESULT - 2012-10-24
Windows 8 32-bit manual-see below - 2012-10-02 Results - yyyy-mm-dd
Windows 8 64-bit manual-see below - 2012-10-02 Results - yyyy-mm-dd
Mac OS X 10.6 RESULT - 2012-10-24 RESULT - 2012-10-24
Mac OS X 10.7 RESULT - 2012-10-24 RESULT - 2012-10-24
Mac OS X 10.8 RESULT - 2012-10-24 RESULT - 2012-10-24
Ubuntu 12.04 32-bit RESULT - 2012-10-24 RESULT - 2012-10-24
Ubuntu 12.04 64-bit RESULT - 2012-10-24 RESULT - 2012-10-24

Windows 8: manual spotcheck until automation in place

  • RELEASETEST
    • RESULT - Firefox 16.0.1 en-US (partial)
    • RESULT - Firefox 13.0 de (complete)
  • RELEASE
    • RESULT - Firefox 16.0.1 en-US (partial)
    • RESULT - Firefox 12.0 fr (complete)
Issues

Issues

Blockers

Full Query
ID Summary Priority Status
793126 crash with error message: "Failed to create temporary texture in system memory. Error code: 2147942414", while navigating street view on Google MapsGL P3 RESOLVED

1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);


Non-Blockers

Full Query
ID Summary Priority Status
786655 On flash content the context menu does not own focus -- RESOLVED
786970 Windowed plugin content gets displayed out of it's container when scrolling the browser window -- RESOLVED
786972 Can't replay the same HTML5 youtube video by clicking on the progress bar after when it ends -- RESOLVED
787056 MapsGL maps.google.com renders blurry on machines Chrome doesn't allow MapsGL enabling on P3 REOPENED
789463 [Vimeo] Cannot seek video using the scrubber when in couch mode -- RESOLVED
797332 Characters typed in twitter's Compose new tweet text box aren't displayed after reaching the last row -- RESOLVED

6 Total; 1 Open (16.67%); 5 Resolved (83.33%); 0 Verified (0%);


Previously Known

Full Query
ID Summary Priority Status
78414 Application shortcut keys (keyboard commands such as f11, ctrl+t, ctrl+r) fail to operate when plug-in (flash, acrobat, quicktime) has focus P3 RESOLVED
667399 RealPlayer G2 LiveConnect plugin crash -- RESOLVED
675645 Firefox freezes when using Pop out while in Fullscreen mode on Youtube P3 RESOLVED
686782 GFX glitches on windows 8 (with D2D) -- RESOLVED
692446 Firefox prompts to install windows media player plugin even though it is already installed -- RESOLVED
717178 Printing stops after 73%, and browser hangs up -- VERIFIED
747909 Youtube.com In Full Screen mode - copy actions are not allowed on HTML5 videos -- RESOLVED
768375 crash in nsDocShell::RestoreFromHistory -- RESOLVED
775082 Video controls disappear after dragging the tabs -- RESOLVED
777051 Clicking "Replay" button on HTML5 Youtube Videos makes the video freeze at the beginning (and fire "ASSERTION: Seek target should lie inside the first audio block after seek [...] nsBuiltinDecoderStateMachine.cpp, line 1883") -- RESOLVED
778819 Toggle Fullscreen in context menu stops workng Flash 11.3.300.268 -- RESOLVED
779492 The youtube.com Flash pop out video player is opened in the background -- RESOLVED
781190 [adbe 3309208] Video images freeze with the latest Flash 11.4.400.252 -- RESOLVED
785754 Buttons on some sites are not rendered right and are like half cut off -- VERIFIED
786237 Babylon toolbar is empty after installation -- RESOLVED
786655 On flash content the context menu does not own focus -- RESOLVED
787056 MapsGL maps.google.com renders blurry on machines Chrome doesn't allow MapsGL enabling on P3 REOPENED
788423 [adbe 3612667] Flash hang in F_1152915508 on dropbox.com -- RESOLVED
791195 Session is not restored after Firefox crash -- RESOLVED
792422 report-a-bug popup div on youtube - page jumping issue -- NEW
794924 Choppy fading out for Quicktime videos -- RESOLVED

21 Total; 2 Open (9.52%); 17 Resolved (80.95%); 2 Verified (9.52%);

Retrieved from "https://wiki.allizom.org/index.php?title=Releases/Firefox_16/Test_Plan&oldid=482738"