MozillaWiki

QA/improve permission prompt UI: Difference between revisions

Page Discussion

QA/improve permission prompt UI (view source)

Revision as of 06:51, 6 April 2017

4,850 bytes added ,  6 April 2017
→‎Checklist
 
(24 intermediate revisions by the same user not shown)
Line 54: Line 54:


In the scope of current testing are following permissions:
In the scope of current testing are following permissions:
* Geolocation
 
* Camera  
* Camera  
* Microphone
* Microphone
* Screen Sharing
* Screen/Window Sharing
* Desktop Notifications
* Desktop Notifications
* Pointer Lock ???
* Geolocation
* Offline Storage ???
* Offilne Storage
* Pop-up Notifications
* Combination between the ones mentioned above
* Combination between the ones mentioned above
=== Out of Scope ===
???


= Requirements for testing =
= Requirements for testing =
== Environments ==
== Environments ==
Full testing will be performed on the following OSes:
Full testing will be performed on the following OSes:
* Windows 10, Windows 7
* Windows 10 x64, Windows 7 x32
* Mac OS X 10.11
* Mac OS X 10.10, Mac OS X 10.11 and Mac OS X 10.12
* Ubuntu 16.04
* Ubuntu 16.04


Line 102: Line 100:
! Ref !! Function !! Test Objective !! Test Type !! Risk Assessment and Coverage(RAC) !! Owners  
! Ref !! Function !! Test Objective !! Test Type !! Risk Assessment and Coverage(RAC) !! Owners  
|-
|-
| 1 || Improve permission prompt UI display || The new visual design is displayed and the notification functionality is correct || Manual || RAC-1, RAC-4 || Eng Team
| TO-1 || Permission prompt for Camera || The new visual design is displayed and the notification functionality is correct || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-2 || Permission prompt for Camera functionality || The basic functions(Allowed, Allowed temporarily, Blocked and Blocked temporarily) work properly || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-3 || Permission prompt for Microphone|| The new visual design is displayed and the notification functionality is correct || Manual || RAC-1, RAC-2 || Eng Team
|-
|-
| 2 || Improve permission prompt UI functionality || The basic functions(Save-don't save, Share-don't share, Allow-don't allow and always remember my decision) all work properly || Manual || RAC-1 || Eng Team
| TO-4 || Permission prompt for Microphone functionality || The basic functions(Allowed, Allowed temporarily, Blocked and Blocked temporarily) work properly || Manual || RAC-1, RAC-2 || Eng Team
|-
|-
| 3 || Different permission prompt types || Validate that new permission prompt design works correctly on different permission prompt types || Manual || RAC-2 || Eng Team
| TO-5 || Permission prompt for Pointer lock|| The new visual design is displayed and the notification functionality is correct || Manual || RAC-1, RAC-2 || Eng Team
|-
|-
| 4 || Performance is acceptable in case of multiple tabs and doesn't crash the browser || The objective the test is trying to demonstrate that new permission prompt design works correctly when on multiple tabs or windows || Manual || RAC-3 || Eng Team
| TO-6 || Permission prompt for Pointer lock functionality || The basic functions(Allowed, Allowed temporarily, Blocked and Blocked temporarily) work properly || Manual || RAC-1, RAC-2 || Eng Team
|-
|-
| 5 || Accessibility || Testing for accessibility includes:  
| TO-7 || Permission prompt for Notifications|| The new visual design is displayed and the notification functionality is correct || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-8 || Permission prompt for Notification functionality|| The basic functions(Allowed, Allowed temporarily, Blocked and Blocked temporarily) work properly || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-9 || Permission prompt for Location|| The new visual design is displayed and the notification functionality is correct || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-10 || Permission prompt for Location functionality|| The basic functions(Allowed, Allowed temporarily, Blocked and Blocked temporarily) work properly || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-11 || Permission prompt for Popups|| The new visual design is displayed and the notification functionality is correct || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-12 || Permission prompt for Popups functionality|| The basic functions(Allowed, Blocked and Show x blocked pop-up) work properly || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-13 || Clearing Permissions || Validate that permission prompt  set can be cleared correctly || Manual || RAC-1, RAC-2, RAC-3 || Eng Team
|-
| TO-14 || Combinations of different permission prompt types || Validate that new permission prompt design works correctly for multiple permission prompt || Manual || RAC-1, RAC-2, RAC-5 || Eng Team
|-
| TO-15 || Performance is acceptable in case of multiple tabs and doesn't crash the browser || The objective the test is trying to demonstrate that new permission prompt design works correctly when on multiple tabs or windows || Manual || RAC-4 || Eng Team
|-
| TO-16 || Accessibility || Testing for accessibility includes:  
- testing using high Contract themes <br />
- testing using high Contract themes <br />
- Screen Reader validation<br />
- Screen Reader validation<br />
- Keyboard-only   
- Keyboard-only   
|| Manual || RAC-1 || Eng Team
|| Manual || RAC-3, RAC-6, RAC-7 || Eng Team
|-
| TO-17 || Localization || RTL and strings in general (contextual warning) || Manual || RAC-8 || Eng Team
|}
|}


Line 123: Line 145:
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score  
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score  
|-
|-
| RAC-1 || Mouse and keyboard actions may not be recognized || TO-1 || 2-Moderate || 1-Unlikely || 3-High || 6
| RAC-1 || Refactoring of permissions prompt UI might affect Firefox base functionality  || TO-1 -> To-14 || 3-High|| 3-Almost Certain|| 3-High || 27
|-
| RAC-2 || Basic permission functionality might get broken || TO-1 -> To-14  || 3- High|| 3-Almost Certain|| 3-High || 27
|-
| RAC-3 || Mouse and keyboard actions may not be recognized || TO-13, TO-16 || 2-Moderate || 1-Unlikely || 3-High || 6
|-
|-
| RAC-2 || Some permission prompt types are not recognized and functional || TO-3 || 3- High|| 2-Possible || 3-High || 18
| RAC-4 || Possible performance issues with multiple tabs for different permission prompt set || TO-15 || 2-Moderate || 2-Possible || 3-High || 12
|-
|-
| RAC-3 || Possible performance issues with multiple tabs || TO-4 || 2-Moderate || 2-Possible || 3-High || 12
| RAC-5 || Possible delay for the permission prompt when accessing a site || TO-14 || 3-High || 2-Possible || 3-High || 18
|-
|-
| RAC-4 || After visiting for the first time a tab containing a permission prompt, the permission prompt starts with a delay || TO-1 || 3-High || 3-Almost Certain || 3-High || 27
| RAC-6 || Backround colors and colors in general are visible and readable under high contrast  || TO-16  || 3-High || 3-Almost Certain|| 2-Medium || 18
|-
|-
| RAC-7 || Screen reader on the warning message? || TO-16 || 1- Low || 2-Possible || 3-High || 6
|-
| RAC-8 || RTL  || TO-17  || 1- Low || 2-Possible || 2-Medium || 4
|}
|}


Line 142: Line 171:


'''Impact Score Breakdown:'''  
'''Impact Score Breakdown:'''  
* An impact value of 1 or 2 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 4, 8 or 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.


Line 159: Line 188:
|-
|-
| Start project  
| Start project  
|style="text-align:center;" | ||  
|style="text-align:center;" | 2017.01.04 ||  
|-
|-
| Study documentation/specs received from developers
| Study documentation/specs received from developers
|style="text-align:center;" |  2017.01.06 || 2017.01.07
|style="text-align:center;" |  2017.01.05
|style="text-align:center;" | 2017.01.12
|-
|-
| QA - Test plan creation  
| QA - Test plan creation  
|style="text-align:center;" |  2017.01.06 || 2017.01.07
|style="text-align:center;" |  2017.01.06  
|style="text-align:center;" | 2017.01.13
|-
|-
| QA - Test cases/Env preparation  
| QA - Test cases/Env preparation  
|style="text-align:center;" | ||  
|style="text-align:center;" | 2017.01.10 ||  
|-
|-
| QA - Nightly Testing  
| QA - Nightly Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | 2017.01.12
|style="text-align:center;" | 2017.01.23
|-
|-
| QA - Aurora Testing
| QA - Aurora Testing
|style="text-align:center;" | ||
|style="text-align:center;" | 2017.01.30
|style="text-align:center;" | 2017.03.06
|-
|-
| QA - Beta Testing  
| QA - Beta Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | 2017.03.06
|style="text-align:center;" | 2017.03.31
|-
|-
| Release Date  
| Release Date  
Line 200: Line 234:
= Status =  
= Status =  
== Overview ==
== Overview ==
  Track the dates and build number where feature was released to Nightly
  First landed in Nightly 53, and we start testing 01/12/2017, with build [http://archive.mozilla.org/pub/firefox/nightly/2017/01/2017-01-12-03-03-01-mozilla-central/ Nightly link]
  Track the dates and build number where feature was merged to Aurora
  Landed on Aurora 53, on 01/23/2017, with build [http://archive.mozilla.org/pub/firefox/nightly/2017/01/2017-01-23-08-03-55-mozilla-aurora/ Aurora link]
  Track the dates and build number where feature was merged to Release/Beta
  Landed on Beta channel, with Beta 53 on 03/06/2017, with build [http://archive.mozilla.org/pub/firefox/candidates/53.0b1-candidates/ Beta 53 - b1]


== Risk analysis ==
Identify the high-risk assumptions
Identify existing bugs on the feature with high risk
Identify if other areas are affected by the fix
= References =
= References =
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1282768 1282768] - Convert permission notifications to use two buttons instead of the drop-down menu
* List and links for specs
* List and links for specs
   Permission prompt specs - ???
   - [https://docs.google.com/document/d/143nEfWfIvFZD2-pFqE8GD85inFjFIhE2J8QeegQufc0/edit# Control Center V2 specs]
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1282768 1282768]
  - [https://docs.google.com/document/d/1MaooflGxc2E9dcNVvz0AwYoTqySfEQOAhOpFNDsJBLs/edit# User research for Permission Prompt Redesign and Insecure Password UI]
   
  - [https://mozilla.invisionapp.com/share/9J7RMC2G7#/screens/170569580 feature samples]
* User Story Used
- [https://bugzilla.mozilla.org/show_bug.cgi?id=1188147 1188147] - [userstory] Notification of Single Permission
- [https://bugzilla.mozilla.org/show_bug.cgi?id=1188118 1188118] - [userstory] ID Block: Control Center "i" anchor icon
- [https://bugzilla.mozilla.org/show_bug.cgi?id=1188152 1188152] - [userstory] Notification when another request is made (multiple state)
- [https://bugzilla.mozilla.org/show_bug.cgi?id=1188355 1188355] - [userstory] CC: Site Permissions on main panel
  - [https://bugzilla.mozilla.org/show_bug.cgi?id=1188472 1188472] - [userstory] ID Block: Permission icon when blocked
 
= Testcases =  
= Testcases =  
== Overview ==
Summary of testing scenarios


== Test Areas ==
== Test Areas ==
Line 257: Line 292:
|-
|-
| RTL build testing   
| RTL build testing   
|style="text-align:center;" | No ||
|style="text-align:center;" | No||
|-
|-
| '''Help/Support'''  
| '''Help/Support'''  
Line 282: Line 317:
|-
|-
| Affects first-run or onboarding     
| Affects first-run or onboarding     
|style="text-align:center;" | No || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|style="text-align:center;" | N/A || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|-
|-
| Does this affect partner builds? Partner build testing   
| Does this affect partner builds? Partner build testing   
|style="text-align:center;" | ??? || yes/no options, add comment with details about who will lead testing
|style="text-align:center;" | N/A || yes/no options, add comment with details about who will lead testing


|-
|-
Line 292: Line 327:
|-
|-
| Enterprise administration   
| Enterprise administration   
|style="text-align:center;" | No ||  
|style="text-align:center;" | N/A ||  
|-
|-
| Network proxies/autoconfig   
| Network proxies/autoconfig   
|style="text-align:center;" | No ||  
|style="text-align:center;" | N/A ||  
|-
|-
| ESR behavior changes   
| ESR behavior changes   
|style="text-align:center;" | No ||  
|style="text-align:center;" | N/A ||  
|-
|-
| Locked preferences   
| Locked preferences   
|style="text-align:center;" | No ||
|style="text-align:center;" | N/A ||


|-
|-
Line 308: Line 343:
|-
|-
| Temporary or permanent telemetry monitoring   
| Temporary or permanent telemetry monitoring   
|style="text-align:center;" | ??? || List of error conditions to monitor
|style="text-align:center;" | N/A || List of error conditions to monitor
|-
|-
| Telemetry correctness testing   
| Telemetry correctness testing   
|style="text-align:center;" | ??? ||  
|style="text-align:center;" | N/A ||  
|-
|-
| Server integration testing   
| Server integration testing   
|style="text-align:center;" | ??? ||  
|style="text-align:center;" | No ||  
|-
|-
| Offline and server failure testing   
| Offline and server failure testing   
|style="text-align:center;" | ??? ||
|style="text-align:center;" | No ||
|-
|-
| Load testing   
| Load testing   
|style="text-align:center;" | ??? ||
|style="text-align:center;" | No ||


|-
|-
Line 333: Line 368:
|-
|-
| Permissions   
| Permissions   
|style="text-align:center;" | No ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Testing with existing/popular addons
| Testing with existing/popular addons
Line 356: Line 391:
|-
|-
| Testing against target sites   
| Testing against target sites   
|style="text-align:center;" | ??? ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Survey of many sites for compatibility   
| Survey of many sites for compatibility   
|style="text-align:center;" | ??? ||  
|style="text-align:center;" | Yes ||  


|-
|-
Line 376: Line 411:


== Test suite ==
== Test suite ==
  Full Test suite - in TestRail at [https://testrail.stage.mozaws.net link]
  Full Test suite - in TestRail at [https://testrail.stage.mozaws.net/index.php?/suites/view/423&group_by=cases:section_id&group_order=asc link]


= Bug Work =
= Bug Work =
Line 414: Line 449:
'''Nightly testing'''<br />
'''Nightly testing'''<br />


List of OSes that will be covered by testing<br />
*List of OSes that will be covered by testing<br />
** Windows 7 x64
** Ubuntu 16.04 x64
** Mac OS X 10.11 (El Capitan)
*Link for the tests run
*Link for the tests run
** Daily Smoke, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
** Full Test suite - [https://testrail.stage.mozaws.net/index.php?/runs/view/1949&group_by=cases:section_id&group_order=asc Nightly tests]
** Full Test suite, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
*This feature was treated as bug work, so no sign-off email was sent
** Regression Test suite, if needed/available


'''Merge to Aurora Sign-off'''
'''Merge to Aurora Sign-off'''<br />
List of OSes that will be covered by testing<br />
*List of OSes that will be covered by testing<br />
** Windows 10 x64
** Ubuntu 16.04 x64
** Mac OS X 10.12 (Sierra)
*Link for the tests run
*Link for the tests run
** Full Test suite
** Full Test suite - [https://testrail.stage.mozaws.net/index.php?/runs/view/2470&group_by=cases:section_id&group_order=asc Aurora tests]
*This feature was treated as bug work, so no sign-off email was sent
 
'''Merge to Beta/Pre-Release Sign-off'''
*List of OSes that will be covered by testing<br />
** Windows 10 x64
** Windows 7 x32
** Ubuntu 16.04 x64
** Mac OS X 10.12 (Sierra)
*Link for the tests run
** Full Test suite - [https://testrail.stage.mozaws.net/index.php?/runs/view/3335&group_by=cases:section_id&group_order=asc Beta test runs]


== Checklist ==
== Checklist ==
Line 431: Line 481:
|-
|-
|  Testing Prerequisites (specs, use cases)  
|  Testing Prerequisites (specs, use cases)  
| style="text-align:center;" |   
| style="text-align:center;" |  {{mdone|}}
| style="text-align:center;" |  
| style="text-align:center;" |  
|-
|-
|  Testing Infrastructure setup  
|  Testing Infrastructure setup  
|style="text-align:center;" |   ||  
|style="text-align:center;" | {{mdone|}}  ||  
|-
|-
|  Test Plan Creation  
|  Test Plan Creation  
| style="text-align:center;" |   ||  
| style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  Test Cases Creation  
|  Test Cases Creation  
|style="text-align:center;" |   ||  
|style="text-align:center;" | {{mdone|}}  ||  
|-
|-
|  Full Functional Tests Execution  
|  Full Functional Tests Execution  
|style="text-align:center;" |   
|style="text-align:center;" |  {{mdone|}}
|style="text-align:center;" |   
|style="text-align:center;" |   
|-
|-
|  Automation Coverage ||
|  Automation Coverage  
|style="text-align:center;" |  
|style="text-align:center;" | {{mdone|}}
|style="text-align:center;" |
|-
|-
|  Performance Testing  
|  Performance Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  All Defects Logged || ||  
|  All Defects Logged  
|style="text-align:center;" | {{mdone|}}||  
|-
|-
|  Critical/Blockers Fixed and Verified || ||  
|  Critical/Blockers Fixed and Verified  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  Metrics/Telemetry||  
|  Metrics/Telemetry||  
|style="text-align:center;" |  
|style="text-align:center;" |  
|-
|-
|  QA Signoff  - Nightly Release||  
|  QA Signoff  - Nightly Release
|style="text-align:center;" | Email to be sent  
|style="text-align:center;" | N/A
|style="text-align:center;" | Was treated as bug work - no sign off email was sent.
|-
|-
|  QA Aurora - Full Testing  
|  QA Aurora - Full Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  QA Signoff  - Aurora Release ||  
|  QA Signoff  - Aurora Release  
|style="text-align:center;"| Email to be sent  
|style="text-align:center;" | N/A
|style="text-align:center;"| Was treated as bug work - no sign off email was sent.
|-
|-
|  QA Beta - Full Testing
|  QA Beta - Full Testing
|style="text-align:center;" | ||  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  QA Signoff  - Beta Release ||  
|  QA Signoff  - Beta Release ||  
|style="text-align:center;" | Email to be sent  
|style="text-align:center;" | Email to be sent  
|}
|}
Brindusa.tot
533

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/1159383...1167677"