MozillaWiki

QA/improve permission prompt UI: Difference between revisions

Page Discussion

QA/improve permission prompt UI (view source)

Revision as of 06:51, 6 April 2017

5,447 bytes added ,  6 April 2017
→‎Checklist
 
(28 intermediate revisions by the same user not shown)
Line 24: Line 24:


== Scope ==
== Scope ==
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
This wiki details the testing that will be performed by the project team for the Permissions Prompt Improvements project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document what and how we shall test:
* What will be tested
* Permissions are listed and correctly managed for devices actively streaming
* How testing will be performed
* The area involving includes every permission doorhanger used in Firefox, with the exception of the plugin prompt.
* Behavior of the permission prompt when the resources (audio/video/sharing) are called from a sub-frame
* Verify permission prompt on secure and unsecured sites.


== Ownership ==
== Ownership ==
'''<big>Development Team</big>''':<br />  
'''<big>Development Team</big>''':<br />  
Manager: [mailto:pastithas@mozilla.com Panagiotis Astithas] (irc: past)<br />
Product Manager: [mailto:pdolanjski@mozilla.com Peter Dolanjski] (irc: pdol)<br />
Project Manager: [mailto:elancaster@mozilla.com Erin Lancaster] (irc: elan)<br />
Engineering Manager: [mailto:pastithas@mozilla.com Panagiotis Astithas] (irc: past)<br />
UX Lead: [mailto:psackl@mozilla.com Philipp Sackl] (irc: phlsa)<br />


'''<big>Engineering Team</big> - Nightly Testing''':<br />  
'''<big>Engineering Team</big> - Nightly Testing''':<br />  
Line 49: Line 54:


In the scope of current testing are following permissions:
In the scope of current testing are following permissions:
* Camera
* Microphone
* Screen/Window Sharing
* Desktop Notifications
* Geolocation
* Geolocation
* Audio
* Offilne Storage
* Video
* Pop-up Notifications
* Screen or Window Sharing ??
* Desktop Notifications
* Pointer Lock
* Offline Storage
* Combination between the ones mentioned above
* Combination between the ones mentioned above
=== Out of Scope ===
???


= Requirements for testing =
= Requirements for testing =
== Environments ==
== Environments ==
Full testing will be performed on the following OSes:
Full testing will be performed on the following OSes:
* Windows 10, Windows 7
* Windows 10 x64, Windows 7 x32
* Mac OS X 10.11
* Mac OS X 10.10, Mac OS X 10.11 and Mac OS X 10.12
* Ubuntu 16.04
* Ubuntu 16.04


Line 97: Line 100:
! Ref !! Function !! Test Objective !! Test Type !! Risk Assessment and Coverage(RAC) !! Owners  
! Ref !! Function !! Test Objective !! Test Type !! Risk Assessment and Coverage(RAC) !! Owners  
|-
|-
| 1 || Improve permission prompt UI display || The new visual design is displayed and the notification functionality is correct || Manual || RAC-1, RAC-4 || Eng Team
| TO-1 || Permission prompt for Camera || The new visual design is displayed and the notification functionality is correct || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-2 || Permission prompt for Camera functionality || The basic functions(Allowed, Allowed temporarily, Blocked and Blocked temporarily) work properly || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-3 || Permission prompt for Microphone|| The new visual design is displayed and the notification functionality is correct || Manual || RAC-1, RAC-2 || Eng Team
|-
|-
| 2 || Improve permission prompt UI functionality || The basic functions(Save-don't save, Share-don't share, Allow-don't allow and always remember my decision) all work properly || Manual || RAC-1 || Eng Team
| TO-4 || Permission prompt for Microphone functionality || The basic functions(Allowed, Allowed temporarily, Blocked and Blocked temporarily) work properly || Manual || RAC-1, RAC-2 || Eng Team
|-
|-
| 3 || Different permission prompt types || Validate that new permission prompt design works correctly on different permission prompt types || Manual || RAC-2 || Eng Team
| TO-5 || Permission prompt for Pointer lock|| The new visual design is displayed and the notification functionality is correct || Manual || RAC-1, RAC-2 || Eng Team
|-
|-
| 4 || Performance is acceptable in case of multiple tabs and doesn't crash the browser || The objective the test is trying to demonstrate that new permission prompt design works correctly when on multiple tabs or windows || Manual || RAC-3 || Eng Team
| TO-6 || Permission prompt for Pointer lock functionality || The basic functions(Allowed, Allowed temporarily, Blocked and Blocked temporarily) work properly || Manual || RAC-1, RAC-2 || Eng Team
|-
|-
| 5 || Accessibility || Testing for accessibility includes:  
| TO-7 || Permission prompt for Notifications|| The new visual design is displayed and the notification functionality is correct || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-8 || Permission prompt for Notification functionality|| The basic functions(Allowed, Allowed temporarily, Blocked and Blocked temporarily) work properly || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-9 || Permission prompt for Location|| The new visual design is displayed and the notification functionality is correct || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-10 || Permission prompt for Location functionality|| The basic functions(Allowed, Allowed temporarily, Blocked and Blocked temporarily) work properly || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-11 || Permission prompt for Popups|| The new visual design is displayed and the notification functionality is correct || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-12 || Permission prompt for Popups functionality|| The basic functions(Allowed, Blocked and Show x blocked pop-up) work properly || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-13 || Clearing Permissions || Validate that permission prompt  set can be cleared correctly || Manual || RAC-1, RAC-2, RAC-3 || Eng Team
|-
| TO-14 || Combinations of different permission prompt types || Validate that new permission prompt design works correctly for multiple permission prompt || Manual || RAC-1, RAC-2, RAC-5 || Eng Team
|-
| TO-15 || Performance is acceptable in case of multiple tabs and doesn't crash the browser || The objective the test is trying to demonstrate that new permission prompt design works correctly when on multiple tabs or windows || Manual || RAC-4 || Eng Team
|-
| TO-16 || Accessibility || Testing for accessibility includes:  
- testing using high Contract themes <br />
- testing using high Contract themes <br />
- Screen Reader validation<br />
- Screen Reader validation<br />
- Keyboard-only   
- Keyboard-only   
|| Manual || RAC-1 || Eng Team
|| Manual || RAC-3, RAC-6, RAC-7 || Eng Team
|-
| TO-17 || Localization || RTL and strings in general (contextual warning) || Manual || RAC-8 || Eng Team
|}
|}


Line 118: Line 145:
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score  
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score  
|-
|-
| RAC-1 || Mouse and keyboard actions may not be recognized || TO-1 || 2-Moderate || 1-Unlikely || 3-High || 6
| RAC-1 || Refactoring of permissions prompt UI might affect Firefox base functionality  || TO-1 -> To-14 || 3-High|| 3-Almost Certain|| 3-High || 27
|-
| RAC-2 || Basic permission functionality might get broken || TO-1 -> To-14  || 3- High|| 3-Almost Certain|| 3-High || 27
|-
| RAC-3 || Mouse and keyboard actions may not be recognized || TO-13, TO-16 || 2-Moderate || 1-Unlikely || 3-High || 6
|-
|-
| RAC-2 || Some permission prompt types are not recognized and functional || TO-3 || 3- High|| 2-Possible || 3-High || 18
| RAC-4 || Possible performance issues with multiple tabs for different permission prompt set || TO-15 || 2-Moderate || 2-Possible || 3-High || 12
|-
|-
| RAC-3 || Possible performance issues with multiple tabs || TO-4 || 2-Moderate || 2-Possible || 3-High || 12
| RAC-5 || Possible delay for the permission prompt when accessing a site || TO-14 || 3-High || 2-Possible || 3-High || 18
|-
|-
| RAC-4 || After visiting for the first time a tab containing a permission prompt, the permission prompt starts with a delay || TO-1 || 3-High || 3-Almost Certain || 3-High || 27
| RAC-6 || Backround colors and colors in general are visible and readable under high contrast  || TO-16  || 3-High || 3-Almost Certain|| 2-Medium || 18
|-
|-
| RAC-7 || Screen reader on the warning message? || TO-16 || 1- Low || 2-Possible || 3-High || 6
|-
| RAC-8 || RTL  || TO-17  || 1- Low || 2-Possible || 2-Medium || 4
|}
|}


Line 137: Line 171:


'''Impact Score Breakdown:'''  
'''Impact Score Breakdown:'''  
* An impact value of 1 or 2 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 4, 8 or 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.


Line 154: Line 188:
|-
|-
| Start project  
| Start project  
|style="text-align:center;" | ||  
|style="text-align:center;" | 2017.01.04 ||  
|-
|-
| Study documentation/specs received from developers
| Study documentation/specs received from developers
|style="text-align:center;" |  2017.01.06 || 2017.01.07
|style="text-align:center;" |  2017.01.05
|style="text-align:center;" | 2017.01.12
|-
|-
| QA - Test plan creation  
| QA - Test plan creation  
|style="text-align:center;" |  2017.01.06 || 2017.01.07
|style="text-align:center;" |  2017.01.06  
|style="text-align:center;" | 2017.01.13
|-
|-
| QA - Test cases/Env preparation  
| QA - Test cases/Env preparation  
|style="text-align:center;" | ||  
|style="text-align:center;" | 2017.01.10 ||  
|-
|-
| QA - Nightly Testing  
| QA - Nightly Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | 2017.01.12
|style="text-align:center;" | 2017.01.23
|-
|-
| QA - Aurora Testing
| QA - Aurora Testing
|style="text-align:center;" | ||
|style="text-align:center;" | 2017.01.30
|style="text-align:center;" | 2017.03.06
|-
|-
| QA - Beta Testing  
| QA - Beta Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | 2017.03.06
|style="text-align:center;" | 2017.03.31
|-
|-
| Release Date  
| Release Date  
Line 195: Line 234:
= Status =  
= Status =  
== Overview ==
== Overview ==
  Track the dates and build number where feature was released to Nightly
  First landed in Nightly 53, and we start testing 01/12/2017, with build [http://archive.mozilla.org/pub/firefox/nightly/2017/01/2017-01-12-03-03-01-mozilla-central/ Nightly link]
  Track the dates and build number where feature was merged to Aurora
  Landed on Aurora 53, on 01/23/2017, with build [http://archive.mozilla.org/pub/firefox/nightly/2017/01/2017-01-23-08-03-55-mozilla-aurora/ Aurora link]
  Track the dates and build number where feature was merged to Release/Beta
  Landed on Beta channel, with Beta 53 on 03/06/2017, with build [http://archive.mozilla.org/pub/firefox/candidates/53.0b1-candidates/ Beta 53 - b1]


== Risk analysis ==
Identify the high-risk assumptions
Identify existing bugs on the feature with high risk
Identify if other areas are affected by the fix
= References =
= References =
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1282768 1282768] - Convert permission notifications to use two buttons instead of the drop-down menu
* List and links for specs
* List and links for specs
   Permission prompt specs - ???
   - [https://docs.google.com/document/d/143nEfWfIvFZD2-pFqE8GD85inFjFIhE2J8QeegQufc0/edit# Control Center V2 specs]
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1282768 1282768]
  - [https://docs.google.com/document/d/1MaooflGxc2E9dcNVvz0AwYoTqySfEQOAhOpFNDsJBLs/edit# User research for Permission Prompt Redesign and Insecure Password UI]
   
  - [https://mozilla.invisionapp.com/share/9J7RMC2G7#/screens/170569580 feature samples]
* User Story Used
- [https://bugzilla.mozilla.org/show_bug.cgi?id=1188147 1188147] - [userstory] Notification of Single Permission
- [https://bugzilla.mozilla.org/show_bug.cgi?id=1188118 1188118] - [userstory] ID Block: Control Center "i" anchor icon
- [https://bugzilla.mozilla.org/show_bug.cgi?id=1188152 1188152] - [userstory] Notification when another request is made (multiple state)
- [https://bugzilla.mozilla.org/show_bug.cgi?id=1188355 1188355] - [userstory] CC: Site Permissions on main panel
  - [https://bugzilla.mozilla.org/show_bug.cgi?id=1188472 1188472] - [userstory] ID Block: Permission icon when blocked
 
= Testcases =  
= Testcases =  
== Overview ==
Summary of testing scenarios


== Test Areas ==
== Test Areas ==
Line 252: Line 292:
|-
|-
| RTL build testing   
| RTL build testing   
|style="text-align:center;" | No ||
|style="text-align:center;" | No||
|-
|-
| '''Help/Support'''  
| '''Help/Support'''  
Line 277: Line 317:
|-
|-
| Affects first-run or onboarding     
| Affects first-run or onboarding     
|style="text-align:center;" | No || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|style="text-align:center;" | N/A || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|-
|-
| Does this affect partner builds? Partner build testing   
| Does this affect partner builds? Partner build testing   
|style="text-align:center;" | ??? || yes/no options, add comment with details about who will lead testing
|style="text-align:center;" | N/A || yes/no options, add comment with details about who will lead testing


|-
|-
Line 287: Line 327:
|-
|-
| Enterprise administration   
| Enterprise administration   
|style="text-align:center;" | No ||  
|style="text-align:center;" | N/A ||  
|-
|-
| Network proxies/autoconfig   
| Network proxies/autoconfig   
|style="text-align:center;" | No ||  
|style="text-align:center;" | N/A ||  
|-
|-
| ESR behavior changes   
| ESR behavior changes   
|style="text-align:center;" | No ||  
|style="text-align:center;" | N/A ||  
|-
|-
| Locked preferences   
| Locked preferences   
|style="text-align:center;" | No ||
|style="text-align:center;" | N/A ||


|-
|-
Line 303: Line 343:
|-
|-
| Temporary or permanent telemetry monitoring   
| Temporary or permanent telemetry monitoring   
|style="text-align:center;" | ??? || List of error conditions to monitor
|style="text-align:center;" | N/A || List of error conditions to monitor
|-
|-
| Telemetry correctness testing   
| Telemetry correctness testing   
|style="text-align:center;" | ??? ||  
|style="text-align:center;" | N/A ||  
|-
|-
| Server integration testing   
| Server integration testing   
|style="text-align:center;" | ??? ||  
|style="text-align:center;" | No ||  
|-
|-
| Offline and server failure testing   
| Offline and server failure testing   
|style="text-align:center;" | ??? ||
|style="text-align:center;" | No ||
|-
|-
| Load testing   
| Load testing   
|style="text-align:center;" | ??? ||
|style="text-align:center;" | No ||


|-
|-
Line 328: Line 368:
|-
|-
| Permissions   
| Permissions   
|style="text-align:center;" | No ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Testing with existing/popular addons
| Testing with existing/popular addons
Line 351: Line 391:
|-
|-
| Testing against target sites   
| Testing against target sites   
|style="text-align:center;" | ??? ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Survey of many sites for compatibility   
| Survey of many sites for compatibility   
|style="text-align:center;" | ??? ||  
|style="text-align:center;" | Yes ||  


|-
|-
Line 371: Line 411:


== Test suite ==
== Test suite ==
  Full Test suite - in TestRail at [https://testrail.stage.mozaws.net link]
  Full Test suite - in TestRail at [https://testrail.stage.mozaws.net/index.php?/suites/view/423&group_by=cases:section_id&group_order=asc link]


= Bug Work =
= Bug Work =
Line 409: Line 449:
'''Nightly testing'''<br />
'''Nightly testing'''<br />


List of OSes that will be covered by testing<br />
*List of OSes that will be covered by testing<br />
** Windows 7 x64
** Ubuntu 16.04 x64
** Mac OS X 10.11 (El Capitan)
*Link for the tests run
*Link for the tests run
** Daily Smoke, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
** Full Test suite - [https://testrail.stage.mozaws.net/index.php?/runs/view/1949&group_by=cases:section_id&group_order=asc Nightly tests]
** Full Test suite, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
*This feature was treated as bug work, so no sign-off email was sent
** Regression Test suite, if needed/available


'''Merge to Aurora Sign-off'''
'''Merge to Aurora Sign-off'''<br />
List of OSes that will be covered by testing<br />
*List of OSes that will be covered by testing<br />
** Windows 10 x64
** Ubuntu 16.04 x64
** Mac OS X 10.12 (Sierra)
*Link for the tests run
*Link for the tests run
** Full Test suite
** Full Test suite - [https://testrail.stage.mozaws.net/index.php?/runs/view/2470&group_by=cases:section_id&group_order=asc Aurora tests]
*This feature was treated as bug work, so no sign-off email was sent
 
'''Merge to Beta/Pre-Release Sign-off'''
*List of OSes that will be covered by testing<br />
** Windows 10 x64
** Windows 7 x32
** Ubuntu 16.04 x64
** Mac OS X 10.12 (Sierra)
*Link for the tests run
** Full Test suite - [https://testrail.stage.mozaws.net/index.php?/runs/view/3335&group_by=cases:section_id&group_order=asc Beta test runs]


== Checklist ==
== Checklist ==
Line 426: Line 481:
|-
|-
|  Testing Prerequisites (specs, use cases)  
|  Testing Prerequisites (specs, use cases)  
| style="text-align:center;" |   
| style="text-align:center;" |  {{mdone|}}
| style="text-align:center;" |  
| style="text-align:center;" |  
|-
|-
|  Testing Infrastructure setup  
|  Testing Infrastructure setup  
|style="text-align:center;" |   ||  
|style="text-align:center;" | {{mdone|}}  ||  
|-
|-
|  Test Plan Creation  
|  Test Plan Creation  
| style="text-align:center;" |   ||  
| style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  Test Cases Creation  
|  Test Cases Creation  
|style="text-align:center;" |   ||  
|style="text-align:center;" | {{mdone|}}  ||  
|-
|-
|  Full Functional Tests Execution  
|  Full Functional Tests Execution  
|style="text-align:center;" |   
|style="text-align:center;" |  {{mdone|}}
|style="text-align:center;" |   
|style="text-align:center;" |   
|-
|-
|  Automation Coverage ||
|  Automation Coverage  
|style="text-align:center;" |  
|style="text-align:center;" | {{mdone|}}
|style="text-align:center;" |
|-
|-
|  Performance Testing  
|  Performance Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  All Defects Logged || ||  
|  All Defects Logged  
|style="text-align:center;" | {{mdone|}}||  
|-
|-
|  Critical/Blockers Fixed and Verified || ||  
|  Critical/Blockers Fixed and Verified  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  Metrics/Telemetry||  
|  Metrics/Telemetry||  
|style="text-align:center;" |  
|style="text-align:center;" |  
|-
|-
|  QA Signoff  - Nightly Release||  
|  QA Signoff  - Nightly Release
|style="text-align:center;" | Email to be sent  
|style="text-align:center;" | N/A
|style="text-align:center;" | Was treated as bug work - no sign off email was sent.
|-
|-
|  QA Aurora - Full Testing  
|  QA Aurora - Full Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  QA Signoff  - Aurora Release ||  
|  QA Signoff  - Aurora Release  
|style="text-align:center;"| Email to be sent  
|style="text-align:center;" | N/A
|style="text-align:center;"| Was treated as bug work - no sign off email was sent.
|-
|-
|  QA Beta - Full Testing
|  QA Beta - Full Testing
|style="text-align:center;" | ||  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  QA Signoff  - Beta Release ||  
|  QA Signoff  - Beta Release ||  
|style="text-align:center;" | Email to be sent  
|style="text-align:center;" | Email to be sent  
|}
|}
Brindusa.tot
533

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/1158948...1167677"