Releases/Firefox 22/Test Plan

« Firefox 21 | Firefox 22 | Firefox 23 »

Firefox 22 Test Plan

Summary

Milestone	Firefox 22
Status	Currently in Aurora -- merging to Beta on 2013-05-14
Lead	Juan Becerra (irc: juanb)
2nd	Anthony Hughes (irc: ashughes)
Builds	[ftp] [http] [web]

The following is the test plan for Firefox 22 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 22. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Details

The following is a description of the tasks which need to be completed throughout this release.

• 2013.02.18: {{|}} Nightly Merge
• 2013.04.01: {{|}} Aurora Merge
• 2013.05.10: {{|}} Pre-Beta feature sign-offs
• 2013.05.16: {{|}} Beta 1
• 2013.05.24: {{|}} Beta 2
• 2013.05.31: {{|}} Beta 3
• 2013.06.07: {{|}} Beta 4
• 2013.06.14: {{|}} Beta 5
• 2013.06.19: {{|}} Beta 6
• 2013.06.24: {{|}} Release Candidate
• 2013.06.25: {{|}} Final

Nightly

Merge

Functional Automation - result

Update Spotchecks - result

Features

See the Feature Release Tracking wiki for a general overview

Features

See the Feature Release Tracking wiki for a general overview

Feature

Owner

Status

Tests

Endurance Automation

The purpose of this section is to track any potential memory regressions via spikes reported in the dashboard.

What to Do

1. Check the Endurance Charts
2. If you see an anomalous event (ie. spike or dip), get the report for the specific event
3. If the event happens twice in a row, file a bug under Mozmill Tests
4. Notify Anthony Hughes of the issue
5. Record the event in this wiki below

Current Status (Mihaela Velimiroviciu) - Last Updated - yyyy.mm.dd

Bugzilla query error

Query options must be valid JSON.1

Functional Automation

The purpose of this section is to track any potential functional regressions via results reported in the dashboard.

What to Do

1. Check the testrun reports
2. If you see any failing tests, check that there are not any bugs on file for it already
3. If not, file a bug under Mozmill Tests
4. Notify Anthony Hughes of the issue
5. Record the event in this wiki below

Current Status (SoftVision) - Last Updated - yyyy-mm-dd

Bug Triage

The purpose of this section is to triage UNCONFIRMED incoming bugs.

What to Do

1. Check the UNCONFIRMED bugs
2. Test the bugs in the latest Firefox Nightly
3. If you can't reproduce the bug, ask for the reporter to provide more information
4. If you reproduce the bug, add your testing information and mark the bug NEW
5. Notify Anthony Hughes of the NEW issue

Current Status (SoftVision) - Last Updated

Result Spreadsheet

Aurora Merge

Builds	nightlies
Changeset	tip (pushlog)

Checklist

• {{|}} Automated Testing
• {{|}} Update Testing on auroratest
• {{|}} Peer Sign-off
• {{|}} Final sign-off
• {{|}} Update Testing on aurora

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform
Windows XP	result - yyyy-mm-dd
Windows Vista	result - yyyy-mm-dd
Windows 7 32-bit	result - yyyy-mm-dd
Windows 7 64-bit	result - yyyy-mm-dd
Windows 8 32-bit	result - yyyy-mm-dd
Windows 8 64-bit	result - yyyy-mm-dd
Mac OS X 10.6	result - yyyy-mm-dd
Mac OS X 10.7	result - yyyy-mm-dd
Mac OS X 10.8	result - yyyy-mm-dd
Ubuntu 12.04 32-bit	result - yyyy-mm-dd
Ubuntu 12.04 64-bit	result - yyyy-mm-dd

Test Failures (known bugs)

Feature sign off

• Platforms: Windows 8 32bit, Windows 7 64bit, Ubuntu 12.10, Mac 10.8.3.

Feature	Owner	Win32	Win64	Mac	Linux
Mixed Content Blocker	Virgil Dicu	NA	NA	NA	NA
H264/AAC/MP3 support for Windows 7/8	Cornel Ionce	[ON TRACK] X Fails	[ON TRACK] X Fails	NA	NA
Input speech	Simona Marcu	NA	NA	NA	NA
Font inspector	Manuela Muntean	[ON TRACK] X Fails	[ON TRACK] X Fails	[ON TRACK] X Fails	[ON TRACK] X Fails
Block cookies from sites I haven't visited	Bogdan Maris	[ON TRACK] X Fails	[ON TRACK] X Fails	[ON TRACK] X Fails	[ON TRACK] X Fails
Prototype DOM Gamepad API	Alexandra Lucinet	[ON TRACK] X Fails	[ON TRACK] X Fails	[ON TRACK] X Fails	[ON TRACK] X Fails
Notification API	Cornel Ionce	[ON TRACK] X Fails	[ON TRACK] X Fails	[ON TRACK] X Fails	[ON TRACK] X Fails

Notes:

• Input Speech:
  • As per Guilherme Goncalves and Com 51 in Bug 650295, it's not possible to use the feature at this point, a backend to perform the actual speech recognition is still needed.
• Mixed Content blocker:
  • Feature is currently prefed off in both Aurora (Verified with Aurora 22) and Nightly - see Bug 834836. When enabled, the feature will target Nightly 23.

Update Testing

Spotcheck updates from a previous Aurora to Aurora 22.

Platform	Aurora channel
Windows 7 64-bit	result - yyyy-mm-dd
Windows 8 64-bit	result - yyyy-mm-dd
Mac OS X 10.7	result - yyyy-mm-dd
Ubuntu 12.04 64-bit	result - yyyy-mm-dd

Issues

Bug Verifications

• [ON TRACK] bug 629801 Implement HTML5 element (Mihaela Velimiroviciu)
• [ON TRACK] bug 839371 Implement HTML5 element (Mihaela Velimiroviciu)
• [ON TRACK] bug 841948 Flip the pref to enable <input type=range> on Nightly and Aurora (Mihaela Velimiroviciu)
• [ON TRACK] bug 840745 default html5 audio/video player controls should facilitate changing playback rate (Mihaela Velimiroviciu)
• [DROPPED] bug 548763 Show download progress in app icon in Mac OS X Dock (Simona Marcu)
  • pending fix for Bug 848792 (see comment 21)
• [ON TRACK] bug 716140 Multithreaded image decoding (Virgil Dicu)
• [ON TRACK] bug 790882 [New Tab Page] Speculatively open connections for sites when hovering them (Virgil Dicu)
• [ON TRACK] bug 737100 Extend Pointer Lock (Mouse Lock) for non-fullscreen elements (Virgil Dicu)
• [ON TRACK] bug 689623 layout needs to provide information on which images are visible or likely to be visible (Paul Silaghi)
• [ON TRACK] bug 755126 management UI via about:addons (Ioana)
• [ON TRACK] bug 841876 Re-enable flexbox pref (by default) in release builds (Ioana)

Bug Triage

1. VERIFYME: verify the fix and do some exploratory testing around the issue
2. FIXED: triage these bugs and test to ensure the fix is effective
3. QAWANTED: triage these bugs and test to fulfill the QA request
4. UNCONFIRMED: triage these bugs for potential new regressions

QAWANTED

Bugs requiring QA's assistance in debugging (steps, regression range, exploratory testing, verification, etc)

Status as of yyyy-mm-dd:

Pre-beta Feature testing and Sign-off

Feature testing

• Platforms: Windows XP, Windows 7 x64, Ubuntu 11.10, Mac 10.7

Run the Firefox 22.0 Pre-beta Feature Sign off tests in MozTrap and report results below.

Feature

Owner

Win32

Mac

Linux

Notes:

Smoke tests

Run the smoketests in MozTrap and report results below.

Platform	Assignee	Result:
Windows XP	tester	RESULTS - yyyy-mm-dd
Windows 7 64-bit	tester	RESULTS - yyyy-mm-dd
Mac OS X 10.7.4	tester	RESULTS - yyyy-mm-dd
Ubuntu 12.04 32-bit	tester	RESULTS - yyyy-mm-dd

Beta Checklist

The following checklist should be completed before signing off for Release.

Beta 1

• {{ |}} Breakpad End-to-End functionality
• {{ |}} Mac OSX 10.8 Signing
• {{ |}} Add-on compatibility (top AMO, top third-party, toolbars, yandex, avg, norton, yahoo, bandoo, conduit, etc)
• {{ |}} Exploratory testing and bug triage against the flagship feature
• Platforms: Windows 7 32-bit, Windows 8 64-bit, Mac OSX 10.8, Ubuntu 12.04 32-bit

Beta 2

• {{ |}} Breakpad End-to-End functionality
• {{ |}} Stop accepting MD5 as a hash algorithm in signatures
• {{ |}} Plug-in compatibility (Flash, Java, Silverlight, Realplayer, Quicktime, Shockwave, Unity [Mac])
• {{ |}} Firefox Beta testday stressing flagship features and bug triage
• {{ |}} Exploratory testing and bug triage against the flagship feature
• Platforms: Windows 7 64-bit, Windows 8 32-bit, Mac OSX 10.7, Ubuntu 12.04 64-bit

Beta 3

• {{ |}} Breakpad End-to-End functionality
• {{ |}} Mac OSX 32-bit mode
• {{ |}} Web compatibility (search, webmail, banking, video, audio, gaming, news, cms, etc.)
• {{ |}} Exploratory testing and bug triage against the flagship feature
• Platforms: Windows 7 32-bit, Windows Vista, Mac OSX 10.6, Ubuntu 12.04 32-bit

Beta 4

• {{ |}} Breakpad End-to-End functionality
• {{ |}} Verify update applies with root owned files
• {{ |}} Add-on compatibility (top AMO, top third-party, toolbars, yandex, avg, norton, yahoo, bandoo, conduit, etc)
• {{ |}} Exploratory testing and bug triage against the flagship feature
• {{ |}} Firefox Beta testday stressing flagship features and bug triage
• Platforms: Windows 7 64-bit, Windows XP, Mac OSX 10.8, Ubuntu 12.10 32-bit

Beta 5

• {{ |}} Breakpad End-to-End functionality
• {{ |}} What's New Page on Firefox update
• {{ |}} Plug-in compatibility (Flash, Java, Silverlight, Realplayer, Quicktime, Shockwave, Unity [Mac])
• {{ |}} Exploratory testing and bug triage against the flagship feature
• Platforms: Windows 7 32-bit, Windows 8 64-bit, Mac OSX 10.7, Ubuntu 12.04 32-bit

Beta 6

• {{ |}} Breakpad End-to-End functionality
• {{ |}} Mac OSX 10.5 and Windows 2000 de-support (install and update)
• {{ |}} Web compatibility (search, webmail, banking, video, audio, gaming, news, cms, etc.)
• {{ |}} Exploratory testing and bug triage against the flagship feature
• Platforms: Windows 7 64-bit, Windows 8 32-bit, Mac OSX 10.6, Ubuntu 12.04 64-bit, Mac OSX 10.5, Windows 2000

New Issues

Bugzilla query error

Query options must be valid JSON.1

Known Issues

Full Query

ID	Summary	Priority	Status
842566	EXC_BAD_ACCESS (SIGSEGV) after building with DMD	--	NEW

1 Total; 1 Open (100%); 0 Resolved (0%); 0 Verified (0%);