QA/improve permission prompt UI

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date	Version	Author	Description
01/06/2017	1.0	Hani Yacoub	Created first draft

Overview

Purpose

Detail the purpose of this document. For example:

• The test scope, focus areas and objectives
• The test responsibilities
• The test strategy for the levels and types of test for this release
• The entry and exit criteria
• The basis of the test estimates
• Any risks, issues, assumptions and test dependencies
• The test schedule and major milestones
• The test deliverables

Scope

This wiki details the testing that will be performed by the project team for the Permissions Prompt Improvements project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document what and how we shall test:

• Permissions are listed and correctly managed for devices actively streaming
• The area involving includes every permission doorhanger used in Firefox, with the exception of the plugin prompt.
• Behavior of the permission prompt when the resources (audio/video/sharing) are called from a sub-frame
• Verify permission prompt on secure and unsecured sites.

Ownership

Development Team:
Product Manager: Peter Dolanjski (irc: pdol)
Project Manager: Erin Lancaster (irc: elan)
Engineering Manager: Panagiotis Astithas (irc: past)
UX Lead: Philipp Sackl (irc: phlsa)

Engineering Team - Nightly Testing:
Rares Bologa (irc: RaresB) - PM for QA team
Brindusa Tot (irc: brindusat) - leading QA efforts
Hani Yacoub (irc: hani) - QA

Testing summary

Scope of Testing

In Scope

This feature purpose is to convert permission notifications to use two buttons instead of the drop-down menu.

The testing effort for the improve permission prompt UI will be invested on the following areas:

• functionality: basic and advanced functionality to be verified according to the existing requirements;
• usability: the Indicator for device permissions must be easy to use and straightforward;
• accessibility: the Indicator for device permissions can be used with High Contrast Theme, Keyboard only or Screen Reader;

In the scope of current testing are following permissions:

• Camera
• Microphone
• Screen/Window Sharing
• Desktop Notifications
• Geolocation
• Offilne Storage
• Pop-up Notifications
• Combination between the ones mentioned above

Requirements for testing

Environments

Full testing will be performed on the following OSes:

• Windows 10 x64, Windows 7 x32
• Mac OS X 10.10, Mac OS X 10.11 and Mac OS X 10.12
• Ubuntu 16.04

Channel dependent settings (configs) and environment setups

Nightly

Aurora

Beta

Post Beta / Release

Test Strategy

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref	Function	Test Objective	Test Type	Risk Assessment and Coverage(RAC)	Owners
TO-1	Permission prompt for Camera	The new visual design is displayed and the notification functionality is correct	Manual	RAC-1, RAC-2	Eng Team
TO-2	Permission prompt for Camera functionality	The basic functions(Allowed, Allowed temporarily, Blocked and Blocked temporarily) work properly	Manual	RAC-1, RAC-2	Eng Team
TO-3	Permission prompt for Microphone	The new visual design is displayed and the notification functionality is correct	Manual	RAC-1, RAC-2	Eng Team
TO-4	Permission prompt for Microphone functionality	The basic functions(Allowed, Allowed temporarily, Blocked and Blocked temporarily) work properly	Manual	RAC-1, RAC-2	Eng Team
TO-5	Permission prompt for Pointer lock	The new visual design is displayed and the notification functionality is correct	Manual	RAC-1, RAC-2	Eng Team
TO-6	Permission prompt for Pointer lock functionality	The basic functions(Allowed, Allowed temporarily, Blocked and Blocked temporarily) work properly	Manual	RAC-1, RAC-2	Eng Team
TO-7	Permission prompt for Notifications	The new visual design is displayed and the notification functionality is correct	Manual	RAC-1, RAC-2	Eng Team
TO-8	Permission prompt for Notification functionality	The basic functions(Allowed, Allowed temporarily, Blocked and Blocked temporarily) work properly	Manual	RAC-1, RAC-2	Eng Team
TO-9	Permission prompt for Location	The new visual design is displayed and the notification functionality is correct	Manual	RAC-1, RAC-2	Eng Team
TO-10	Permission prompt for Location functionality	The basic functions(Allowed, Allowed temporarily, Blocked and Blocked temporarily) work properly	Manual	RAC-1, RAC-2	Eng Team
TO-11	Permission prompt for Popups	The new visual design is displayed and the notification functionality is correct	Manual	RAC-1, RAC-2	Eng Team
TO-12	Permission prompt for Popups functionality	The basic functions(Allowed, Blocked and Show x blocked pop-up) work properly	Manual	RAC-1, RAC-2	Eng Team
TO-13	Clearing Permissions	Validate that permission prompt set can be cleared correctly	Manual	RAC-1, RAC-2, RAC-3	Eng Team
TO-14	Combinations of different permission prompt types	Validate that new permission prompt design works correctly for multiple permission prompt	Manual	RAC-1, RAC-2, RAC-5	Eng Team
TO-15	Performance is acceptable in case of multiple tabs and doesn't crash the browser	The objective the test is trying to demonstrate that new permission prompt design works correctly when on multiple tabs or windows	Manual	RAC-4	Eng Team
TO-16	Accessibility	Testing for accessibility includes: - testing using high Contract themes - Screen Reader validation - Keyboard-only	Manual	RAC-3, RAC-6, RAC-7	Eng Team
TO-17	Localization	RTL and strings in general (contextual warning)	Manual	RAC-8	Eng Team

Risk Assessment and Coverage

ID	Description / Threat Description	Covered by Test Objective	Magnitude	Probability	Priority	Impact Score
RAC-1	Refactoring of permissions prompt UI might affect Firefox base functionality	TO-1 -> To-14	3-High	3-Almost Certain	3-High	27
RAC-2	Basic permission functionality might get broken	TO-1 -> To-14	3- High	3-Almost Certain	3-High	27
RAC-3	Mouse and keyboard actions may not be recognized	TO-13, TO-16	2-Moderate	1-Unlikely	3-High	6
RAC-4	Possible performance issues with multiple tabs for different permission prompt set	TO-15	2-Moderate	2-Possible	3-High	12
RAC-5	Possible delay for the permission prompt when accessing a site	TO-14	3-High	2-Possible	3-High	18
RAC-6	Backround colors and colors in general are visible and readable under high contrast	TO-16	3-High	3-Almost Certain	2-Medium	18
RAC-7	Screen reader on the warning message?	TO-16	1- Low	2-Possible	3-High	6
RAC-8	RTL	TO-17	1- Low	2-Possible	2-Medium	4

Values:

• Magnitude: 1- Low , 2-Moderate, 3-High

• Probability: 1-Unlikely, 2-Possible, 3-Almost Certain

• Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

• An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
• An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
• An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

Builds

This section should contain links for builds with the feature -

• Links for Nightly builds
• Links for Aurora builds
• Links for Beta builds

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase	Start Date	End Date
Start project	2017.01.04
Study documentation/specs received from developers	2017.01.05	2017.01.12
QA - Test plan creation	2017.01.06	2017.01.13
QA - Test cases/Env preparation	2017.01.10
QA - Nightly Testing	2017.01.12	2017.01.23
QA - Aurora Testing	2017.01.30	2017.03.06
QA - Beta Testing	2017.03.06	2017.03.31
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process	Tool
Test plan creation	Mozilla wiki
Test case creation	TestRail/ Google docs
Test case execution	TestRail
Bugs management	Bugzilla

Status

Overview

First landed in Nightly 53, and we start testing 01/12/2017, with build Nightly link
Landed on Aurora 53, on 01/23/2017, with build Aurora link
Landed on Beta channel, with Beta 53 on 03/06/2017, with build Beta 53 - b1

References

• Meta bug 1282768 - Convert permission notifications to use two buttons instead of the drop-down menu
• List and links for specs

 - Control Center V2 specs
 - User research for Permission Prompt Redesign and Insecure Password UI
 - feature samples

• User Story Used

- 1188147 - [userstory] Notification of Single Permission
- 1188118 - [userstory] ID Block: Control Center "i" anchor icon
- 1188152 - [userstory] Notification when another request is made (multiple state)
- 1188355 - [userstory] CC: Site Permissions on main panel
- 1188472 - [userstory] ID Block: Permission icon when blocked

Testcases

Test Areas

Test Areas	Covered	Details
Private Window	Yes
Multi-Process Enabled	Yes
Multi-process Disabled	Yes
Theme (high contrast)	Yes
UI
Mouse-only operation	Yes
Keyboard-only operation	Yes
Display (HiDPI)	Yes
Interraction (scroll, zoom)	Yes
Usable with a screen reader	Yes	e.g. with NVDA
Usability and/or discoverability testing	Yes	Is this feature user friendly
RTL build testing	No
Help/Support
Help/support interface required	No	Make sure link to support/help page exist and is easy reachable.
Support documents planned(written)	No	Make sure support documents are written and are correct.
Install/Upgrade
Feature upgrades/downgrades data as expected	No
Does sync work across upgrades	No
Requires install testing	No	separate feature/application installation needed (not only Firefox)
Affects first-run or onboarding	N/A	Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
Does this affect partner builds? Partner build testing	N/A	yes/no options, add comment with details about who will lead testing
Enterprise		Raise up the topic to developers to see if they are expecting to work different on ESR builds
Enterprise administration	N/A
Network proxies/autoconfig	N/A
ESR behavior changes	N/A
Locked preferences	N/A
Data Monitoring
Temporary or permanent telemetry monitoring	N/A	List of error conditions to monitor
Telemetry correctness testing	N/A
Server integration testing	No
Offline and server failure testing	No
Load testing	No
Add-ons		If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required?	No
Comprehensive API testing	No
Permissions	Yes
Testing with existing/popular addons	No
Security		Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
3rd-party security review	No
Privilege escalation testing	No
Fuzzing	No
Web Compatibility		depends on the feature
Testing against target sites	Yes
Survey of many sites for compatibility	Yes
Interoperability		depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.	No
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS	No
Interaction of this feature with other browser features	No

Test suite

Full Test suite - in TestRail at link

Bug Work

Tracking bug – meta bug

Bug fix verification

Bug No	Summary	Status	Firefox Verion
123	bug summary	NEW	Nighly 45

Logged bugs

Bug 111111
Bug 211111

Sign off

Criteria

Check list

• All test cases should be executed
• All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

• List of OSes that will be covered by testing
  
  • Windows 7 x64
  • Ubuntu 16.04 x64
  • Mac OS X 10.11 (El Capitan)
• Link for the tests run
  • Full Test suite - Nightly tests
• This feature was treated as bug work, so no sign-off email was sent

Merge to Aurora Sign-off

• List of OSes that will be covered by testing
  
  • Windows 10 x64
  • Ubuntu 16.04 x64
  • Mac OS X 10.12 (Sierra)
• Link for the tests run
  • Full Test suite - Aurora tests
• This feature was treated as bug work, so no sign-off email was sent

Merge to Beta/Pre-Release Sign-off

• List of OSes that will be covered by testing
  
  • Windows 10 x64
  • Windows 7 x32
  • Ubuntu 16.04 x64
  • Mac OS X 10.12 (Sierra)
• Link for the tests run
  • Full Test suite - Beta test runs

Checklist

Exit Criteria	Status	Notes/Details
Testing Prerequisites (specs, use cases)	[DONE]
Testing Infrastructure setup	[DONE]
Test Plan Creation	[DONE]
Test Cases Creation	[DONE]
Full Functional Tests Execution	[DONE]
Automation Coverage		[DONE]
Performance Testing	[DONE]
All Defects Logged	[DONE]
Critical/Blockers Fixed and Verified	[DONE]
Metrics/Telemetry
QA Signoff - Nightly Release	N/A	Email to be sent
QA Aurora - Full Testing	[DONE]
QA Signoff - Aurora Release	N/A	Email to be sent
QA Beta - Full Testing	[DONE]
QA Signoff - Beta Release		Email to be sent