Add-ons/QA/Testplan/OOP-WebExtensions

Revision History

Date	Version	Author	Description
02/03/2017	1.0	Vasilica Mihasca	Created first draft

Overview

With webext-oop, addon code runs in a separate process (bugzil.la/webext-oop). In the far future, addon code may also run in multiple processes.

Purpose

Entry Criteria

• QA has access to all the PRDs, mocks and related documents
• The feature has landed on Nightly

Exit Criteria

• All the bugs against the feature have been triaged
• All the P1/P2 bugs have been fixed
• All the resolved bugs have been verified by QA
• The find/fixed rate is going down over a predefined period of time

Acceptance Criteria

This section broadly outlines when the product is ready to ship

• QA has signed off
• All the required Telemetry is in place
• All info is localized at least for a pre-defined set of locales
• All the necessary PR/blogposts have been sent out

Scope

This section describes what parts of the feature will be tested and what parts won't be.

what's in scope?

• Webextensions installation with e10s enabled
• Webextensions running in their own process
• Regression testing for the top 20 webextensions

what's out of scope?

• Security testing
• Performance testing

Ownership

Dev Lead: []; irc nick
QA Manager: Krupa Raj; irc nick :krupa
QA Lead: Victor Carciu; irc nick :victorc
Webextensions QA: Vasilica Mihasca; irc nick :vasilica_mihasca
Add-ons QA: Valentina Peleski; irc nick :ValentinaV

Requirements for testing

Environments

OSed covered: Windows, Mac OS X, Linux

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref	Function	Test Objective	Evaluation Criteria	Test Type	Owners

Builds

This section should contain links for builds with the feature -

• Links for Nightly builds
• Links for Aurora builds
• Links for Beta builds

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase	Start Date	End Date
Start project
Study documentation/specs received from developers
QA - Test plan creation	02-03-2017
QA - Test cases/Env preparation
QA - Nightly Testing
QA - Aurora Testing
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process	Tool
Test plan creation	Mozilla wiki
Test case creation	[ TestRail] / [Google docs] / [etherpad]
Test case execution	[TestRail]
Bugs management	Bugzilla / Github

Status

Overview

Track the dates and build number where feature was released to Nightly
Track the dates and build number where feature was merged to Aurora
Track the dates and build number where feature was merged to Release/Beta

Risk analysis

Identify the high-risk assumptions
Identify existing bugs on the feature with high risk
Identify if other areas are affected by the fix

References

* List and links for specs
  PRD - 
* bug 1190679 - (webext-oop) Run WebExtensions out of process

Full Query

ID	Priority	Component	Assigned to	Summary	Status	Target milestone
1202478	--	Untriaged	Bill McCloskey [inactive unless it's an emergency] (:billm)	Run WebExtension callbacks asynchronously	RESOLVED	mozilla44
1224105	--	Untriaged	Bill McCloskey [inactive unless it's an emergency] (:billm)	Use a <browser> element for the background page	RESOLVED	mozilla46
1253748	P2	Graphics		Compositor plugin window management should support multiple remote layer trees with plugin windows	RESOLVED	---
1287004	P1	General	David Anderson [:dvander] - inactive, e-mail if emergency	Support remote <browser> elements in windowless browsers	RESOLVED	mozilla54
1287007	--	Untriaged	Rob Wu [:robwu]	Start using ChildAPIManager for some APIs	RESOLVED	mozilla52
1287010	--	Untriaged	Rob Wu [:robwu]	Some API methods need to run locally with OOP webextensions	RESOLVED	mozilla51
1287209	P3	Frontend	Kris Maglione [:kmag]	Popup and options UI resizing code is not compatible with remote browsers	RESOLVED	mozilla52
1287210	--	Untriaged	Kris Maglione [:kmag]	setIcon calls with ImageData objects are not cross-process compatible	RESOLVED	---
1287977	P2	Untriaged		Modify E10sUtils.jsm to run WebExtension tab pages in the WebExtensions process	RESOLVED	---
1287978	--	Untriaged		Need to allow <browser> element to choose to run in a WebExtension process	RESOLVED	---
1288276	--	Untriaged	Rob Wu [:robwu]	closeProxyContext is never invoked	RESOLVED	mozilla51
1288279	--	Untriaged	Rob Wu [:robwu]	WebExt contextId must be globally unique, even with multiple processes	RESOLVED	mozilla51
1302702	P1	Developer Tools	Luca Greco [:rpl] [:luca] [:lgreco]	Make WebExtension Addon Debugging oop-compatible	VERIFIED	mozilla55
1305216	P3	General	Rob Wu [:robwu]	Use ChildAPIManager for webNavigation API	RESOLVED	---
1305217	P1	General	Kris Maglione [:kmag]	Make webRequest oop-compatible	RESOLVED	mozilla52
1306110	P2	General	Kris Maglione [:kmag]	Run WebExtension tests with both remote and non-remote APIs	RESOLVED	---
1317101	--	General	Kris Maglione [:kmag]	Run WebExtension tests in both in-process and out-of-process modes	RESOLVED	mozilla53
1320395	--	General	Kris Maglione [:kmag]	Run remote extensions in their own process type	RESOLVED	mozilla53
1334557	P3	Security: Process Sandboxing		Tighten sandboxing for extension content process	NEW	---
1339144	P1	General	Kris Maglione [:kmag]	Use windowless remote <browser> element for background pages	RESOLVED	mozilla54
1353060	P1	Graphics: Layers	Kris Maglione [:kmag]	Remote <browser>s are not visible as children of XUL <popup>s	RESOLVED	mozilla55
1353959	P1	General	Kris Maglione [:kmag]	Need a test webextension to crash the extension process	RESOLVED	---
1355239	P3	General	Rob Wu [:robwu]	We should attempt to recover from an extension process crash.	RESOLVED	120 Branch
1356317	P1	Graphics: Layers	Stephen A Pohl [:spohl]	Support remote layer trees in popups on Windows	VERIFIED	mozilla56
1357486	P1	General	Kris Maglione [:kmag]	Turn on OOP extensions by default on Windows	RESOLVED	mozilla56
1357487	P2	General	Kris Maglione [:kmag]	Turn on OOP extensions by default on Linux	VERIFIED	mozilla63
1357490	--	General	Kris Maglione [:kmag]	Run xpcshell tests in both in-process and OOP modes	RESOLVED	mozilla56
1357729	--	Untriaged		blank browserAction with extensions.webextensions.remote=true	RESOLVED	---
1361661	P1	Telemetry	Georg Fritzsche [:gfritzsche]	Send extension process Telemetry in a separate payload section	RESOLVED	mozilla55
1362457	P1	General	Robert Helmer [:rhelmer]	No crash report generated for extension process crashes	RESOLVED	---
1365660	P3	Frontend	Kris Maglione [:kmag]	Add special handling for popups which will contain remote browsers	RESOLVED	mozilla55
1369559	P5	General		enable autocomplete popup test for remote popups	NEW	---
1375490	P1	Untriaged	Kris Maglione [:kmag]	Alerts do not work correctly in OOP popup browsers	VERIFIED	mozilla57
1379046	--	Graphics: Layers	Kris Maglione [:kmag]	RenderFrameParent inconsistently uses the wrong LayerManager in XUL popups	RESOLVED	mozilla56
1379508	P1	Frontend	Kris Maglione [:kmag]	<select> dropdowns don't work in OOP popup browsers	RESOLVED	mozilla56
1380109	P1	General	Kris Maglione [:kmag]	[oop webextensions] When moving a WebExtension to the panel, it closes the panel when clicked (doesn't open subview)	RESOLVED	mozilla56
1380156	P1	Request Handling	Kris Maglione [:kmag]	Loading temporary an unpacked extension breaks extension page's CSS in OOP Extensions	VERIFIED	mozilla56
1380290	P2	Untriaged	Kris Maglione [:kmag]	Options page can no longer receive messages from the background page	VERIFIED	mozilla56
1380294	P1	General	Kris Maglione [:kmag]	[Firefox Notes] Formatting option tooltips are wrongly displayed on the latest Nightly	VERIFIED	mozilla57
1380597	P1	Frontend	Shane Caraveo (:mixedpuppy)	sidebarAction.setPopup is missing a URL security check	RESOLVED	mozilla57
1380622	P2	Untriaged		[Windows] Visual glitches while opening the webextension panel	RESOLVED	---
1381023	--	General		[Windows] "Screenshots went haywire" error message displayed for any screenshot taken and the Screenshots overlay wrongly remains displayed on root page	VERIFIED	mozilla56
1381086	--	Untriaged		extensions.webextensions.remote: true breaks Containers add-on	RESOLVED	---
1381097	P3	Graphics: Layers	Kris Maglione [:kmag]	Popups with APZ enabled to not repaint correctly after resize	RESOLVED	---
1381321	--	Frontend	Kris Maglione [:kmag]	<select> popups don't work in OOP options browsers	VERIFIED	mozilla56
1381337	--	Frontend	Kris Maglione [:kmag]	Select dropdowns in popups should not be constrained to document viewport	RESOLVED	mozilla56
1381344	P1	Untriaged	Kris Maglione [:kmag]	No context menu for OOP popups opened as standalone panel	VERIFIED	mozilla57
1381810	P3	Untriaged		[Ubuntu] Black borders displayed for webextension panel	RESOLVED	---
1382228	--	Untriaged		Unable to Disable or Remove Awesome Screenshot Plus - Capture add-on	RESOLVED	---
1382271	P1	Request Handling	Kris Maglione [:kmag]	<script>s referencing non-existent moz-extension: scripts never resume parsing with protocol remoting enabled	RESOLVED	---
1384078	P1	Untriaged	Kris Maglione [:kmag]	Enter closes WebExtensions popup	VERIFIED	mozilla57
1385403	P3	Graphics: Layers	Kris Maglione [:kmag]	Turn on OOP extensions by default on Mac OS	VERIFIED	mozilla61
1385548	P2	Frontend	Luca Greco [:rpl] [:luca] [:lgreco]	Confirm/alert/prompt dialogs do not work from options_ui with OOP enabled	VERIFIED	mozilla58
1385736	P2	IPC		How to distinguish between content processes and Extension OOP processes?	RESOLVED	---
1385880	P5	Untriaged	Kris Maglione [:kmag]	Disable/Enable hides extension options	VERIFIED	mozilla57
1387386	P5	General		Long URL breaks the webextension panel while alert pop-up is displayed	UNCONFIRMED	---
1388336	--	Developer Outreach		Layout issue in AdBlock for Firefox details tab	RESOLVED	---
1389496	P3	Compatibility		Slow getHost method in Page Zoom WE	RESOLVED	---
1390276	P3	Add-ons Manager	Mark Striemer [:mstriemer]	Update options_ui to match photon	VERIFIED	mozilla58
1390445	P2	Frontend	Luca Greco [:rpl] [:luca] [:lgreco]	HTML <select> drop-downs are incorrectly positioned in WebExtensions add-on Options page	VERIFIED	mozilla58
1392210	P2	Frontend	Shane Caraveo (:mixedpuppy)	[OOP] onBeforeLinkTraversal is not called for extension frames	VERIFIED	mozilla57
1393150	P1	General	Shane Caraveo (:mixedpuppy)	require e10s if webextension.remote=true	VERIFIED	mozilla57
1394208	--	Frontend		Popup with scrollable content in overflow menu has painting artifacts with OOP enabled	RESOLVED	---
1394807	--	General		Tab pages for OOP extensions don't work when e10s is disbled	RESOLVED	---
1395898	--	Request Handling	Haik Aftandilian [:haik]	[OOP] jar cache incorrectly handled non-existent file paths	VERIFIED	mozilla57
1397210	P3	Frontend		[OOP] Opening sidebar link inconsistencies with new tab active state and container inheritance	NEW	---
1401288	P2	Developer Tools	Luca Greco [:rpl] [:luca] [:lgreco]	Ensure that the WebExtensions "devtools panel" browsers run in the same TabGroup of the other extension pages	RESOLVED	mozilla57
1403965	P2	General	Shane Caraveo (:mixedpuppy)	[OOP] The context menu is displayed in a wrong position after moving extension sidebar to right / left	VERIFIED	mozilla61
1405001	P3	Panning and Zooming		Error processing PAPZCTreeManager::Msg_SetTargetAPZC message when opening dropdown in remote webextension's options panel	NEW	---
1408756	P3	Frontend		Can't drag and drop html elements in extension Options page in Firefox with out-of-process WebExtensions	NEW	---
1411285	P3	General		Pop-up is not working with the OOP active and an error is logged in the browser console	NEW	---
1415860	P5	General		New Tab page cannot grab focus for Out of Process extensions	UNCONFIRMED	---
1417043	P2	Developer Tools	Luca Greco [:rpl] [:luca] [:lgreco]	Drop down menus broken in extension defined devtools panels on Windows	VERIFIED	mozilla60
1418394	P3	Frontend		Popup window created by extension cannot close itself on Windows 10	UNCONFIRMED	---
1418655	P3	Frontend	Rob Wu [:robwu]	browser.tabs.onUpdated status == "complete" fires before tab can receive messages	REOPENED	---
1419285	--	IPC	Makoto Kato [:m_kato]	IME candidate window position is always top-left when using Hatena Bookmark Web Extension	VERIFIED	mozilla59
1422187	P3	Untriaged		The hover action for a browser action popup stays partially highlighted after hover release	RESOLVED	---
1423817	--	General		Caret (text cursor) stuck in place with extensions.webextensions.remote = true	RESOLVED	---
1433879	P3	General		WebExtensions' tab listener handler appears to be too slow (bad performance)	NEW	---
1435959	P2	Developer Tools	Luca Greco [:rpl] [:luca] [:lgreco]	Network monitor for add-on background script doesn't work on Windows	VERIFIED	mozilla60
1560835	P3	General		Extension preferences in about:addons become blank when tab detached into separate window	NEW	---
1827085	P3	General		Run every extension in separate processes	NEW	---

82 Total; 13 Open (15.85%); 46 Resolved (56.1%); 23 Verified (28.05%);

Bug Work

Tracking bug - bug 1190679

Bug fix verification

Logged bugs

Sign off

Criteria

Check list

• All test cases should be executed
• Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
• All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

List of OSes that will be covered by testing

• Link for the tests run
  • Daily Smoke, use template from [link]
  • Full Test suite, use template from [ link]
  • Regression Test suite, if needed/available

Merge to Aurora Sign-off List of OSes that will be covered by testing

• Link for the tests run
  • Full Test suite

Checklist

Exit Criteria	Status	Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation
Test Cases Creation
Full Functional Tests Execution
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
QA Signoff - Nightly Release		Email to be sent
QA Aurora - Full Testing
QA Signoff - Aurora Release		Email to be sent
QA Beta - Full Testing
QA Signoff - Beta Release		Email to be sent