Releases/Firefox 17/Test Plan

« Firefox 16 | Firefox 17 | Firefox 18 »

Firefox 17 Test Plan

Summary

Milestone	Firefox 17
Status	Currently in Aurora -- merging to Beta on October 9, 2012
Lead	Anthony Hughes (irc: ashughes)
2nd	Juan Becerra (irc: juanb)
Builds	[ftp] [http] [web]

The following is the test plan for Firefox 17 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 17. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Details

The following is a description of the tasks which need to be completed throughout this release.

• 2012-07-16: {{|}} Nightly Merge
• 2012-08-27: {{|}} Aurora Merge
• 2012-10-08: {{|}} Pre-Beta feature sign-offs
• 2012-10-11: {{|}} Beta 1
• 2012-10-18: {{|}} Beta 2
• 2012-10-25: {{|}} Beta 3
• 2012-11-01: {{|}} Beta 4
• 2012-11-08: {{|}} Beta 5
• 2012-11-15: {{|}} Beta 6
• 2012-11-19: {{|}} Release Candidate
• 2012-11-20: {{|}} Final

Nightly

Merge

Functional Automation - result

Update Spotchecks - result

Features

See the Feature Release Tracking wiki for a general overview

Features

See the Feature Release Tracking wiki for a general overview

Feature	Owner	Status	Tests
Improve display of location bar results	Virgil Dicu	On track
Firefox Social Integration	Anthony Hughes	On track
Silent Update not now prompt	Ioana Budnar	Release target removed.

Endurance Automation

The purpose of this section is to track any potential memory regressions via spikes reported in the dashboard.

What to Do

1. Check the Charts
2. If you see an anomalous event (ie. spike or dip), get the report for the specific event
3. If the event happens twice in a row, file a bug under Mozmill Tests
4. Notify Anthony Hughes of the issue
5. Record the event in this wiki below

Current Status (Mihaela Velimiroviciu) - Last Updated

Bugzilla query error

Query options must be valid JSON.1

Functional Automation

The purpose of this section is to track any potential functional regressions via results reported in the dashboard.

What to Do

1. Check the testrun reports
2. If you see any failing tests, check that there are not any bugs on file for it already
3. If not, file a bug under Mozmill Tests
4. Notify Anthony Hughes of the issue
5. Record the event in this wiki below

Current Status (Vlad Maniac / Alex Lakatos) - Last Updated

Bug Triage

The purpose of this section is to triage UNCONFIRMED incoming bugs.

What to Do

1. Check the UNCONFIRMED bugs
2. Test the bugs in the latest Firefox Nightly
3. If you can't reproduce the bug, ask for the reporter to provide more information
4. If you reproduce the bug, add your testing information and mark the bug NEW
5. Notify Anthony Hughes of the NEW issue

Current Status (Vlad Ghetiu / Virgil Dicu) - Last Updated

Result Spreadsheet

Aurora Merge

NOTE - OSX 10.5 should not be able to install nor update to Firefox 17

Builds	nightly
Changeset	tip (pushlog)

Checklist

• {{ |}} Automated Testing
• [ON TRACK] Manual Testing
• {{ |}} Update Testing
• {{ |}} Bug Triage
• {{ |}} Peer Sign-off
• {{ |}} Final sign-off sent and updates qualified on Beta channel

Automated Testing

The following are the detailed results of the automated Mozmill test runs.

Platform	PASS: report
Windows XP	PASS - 2012-08-28
Windows Vista	PASS - 2012-08-28
Windows 7 32-bit	PASS - 2012-08-28
Windows 7 64-bit	PASS - 2012-08-28
Windows 8 32-bit	N/A (pending IT)
Windows 8 64-bit	N/A (pending IT)
Mac OS X 10.6	PASS - 2012-08-28
Mac OS X 10.7	PASS - 2012-08-28
Mac OS X 10.8	PASS - 2012-08-28
Ubuntu 12.04 32-bit	PASS - 2012-08-28
Ubuntu 12.04 64-bit	PASS - 2012-08-28

Test Failures (known bugs)

Manual Testing

Operating System	Tester	Plug-in Compatibility	Web Compatibility	OSX 10.5 EOL
Windows XP	Simona	In progress - 2012-08-30	In progress - 2012-08-30	N/A
Windows 7	Ioana	PASS - 2012-08-29	PASS - 2012-08-29	N/A
Windows 8	Paul	result - YYYY-MM-DD	result - YYYY-MM-DD	N/A
Mac OS X	VladG	In progress - 2012-08-30	In progress - 2012-08-30	In progress - 2012-08-30
Ubuntu 12.04	Ioana	PASS - 2012-08-30	In progress - 2012-08-30	N/A

Plug-in Compatibility

Install the following plug-ins and go to websites which use them, looking for rendering issues, functional issues, performance issues, and crashes.

• Adobe Flash Player
  • 11.4 Release (make sure you can autoupdate)
• Microsoft Silverlight
• RealPlayer
• QuickTime

Web Compatibility

Test the following websites for rendering issues, functional issues, performance issues, and crashes

1. Overall Top 10
2. Top by Country (pick a random site from three random countries)
3. Top by Category (pick a random site from three random categories)
4. Webmail clients like gmail, hotmail, and yahoo mail (pay particular attention to any built in web messenger software)
5. Try Google's MapsGL demo
6. Try YouTube's HTML5 demo

OSX 10.5 EOL

Verify the following:

1. Mac OSX 10.5 cannot download Firefox 17 (I'm not sure if this is scoped but try it anyway)
2. Mac OSX 10.5 cannot start Firefox 17
3. Mac OSX 10.5 cannot update to Firefox 17

Feature Sign-offs

• Log in to moztrap.mozilla.org
• Select Run Tests
• Select Product: Desktop Firefox
• Select Versions: 17
• Select Runs: Firefox 17 Aurora Merge Feature sign off

Feature	Owner	Win32	Mac	Linux
Improve display of location bar results	Virgil Dicu	result	result	result
Firefox Social Integration	Anthony Hughes	result	result	result

Details

Report any detailed results or bugs here...

Update Testing

Spotcheck updates from a previous Aurora to Aurora 17.

Platform	Auroratest	Aurora
Windows 7 64-bit	result - YYYY-MM-DD	result - YYYY-MM-DD
Windows 8 64-bit	result - YYYY-MM-DD	result - YYYY-MM-DD
Mac OS X 10.7	result - YYYY-MM-DD	result - YYYY-MM-DD
Ubuntu 12.04 64-bit	result - YYYY-MM-DD	result - YYYY-MM-DD

Issues

Bug Triage

1. VERIFYME (5): verify the fix and do some exploratory testing around the issue
2. FIXED (139): triage these bugs and test to ensure the fix is effective
3. QAWANTED (6): triage these bugs and test to fulfill the QA request
4. UNCONFIRMED (8): triage these bugs for potential new regressions

Results

• VERIFIED: 24/139
• WONTVERIFY: 3/139
• HELPWANTED: 9/139

New Issues

Full Query

ID	Summary	Priority	Status
786970	Windowed plugin content gets displayed out of it's container when scrolling the browser window	--	RESOLVED

1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);

Known Issues

Full Query

ID	Summary	Priority	Status
468855	Bringing up a context-menu on a Flash video player pauses the video	--	RESOLVED
778819	Toggle Fullscreen in context menu stops workng Flash 11.3.300.268	--	RESOLVED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);