QA/improve permission prompt UI: Difference between revisions

From MozillaWiki
< QA
Jump to navigation Jump to search
No edit summary
 
(38 intermediate revisions by 2 users not shown)
Line 24: Line 24:


== Scope ==
== Scope ==
This wiki details the testing that will be performed by the project team for the <project name> project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
This wiki details the testing that will be performed by the project team for the Permissions Prompt Improvements project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document what and how we shall test:
* What will be tested
* Permissions are listed and correctly managed for devices actively streaming
* How testing will be performed
* The area involving includes every permission doorhanger used in Firefox, with the exception of the plugin prompt.
* Behavior of the permission prompt when the resources (audio/video/sharing) are called from a sub-frame
* Verify permission prompt on secure and unsecured sites.


== Ownership ==
== Ownership ==
'''<big>Development Team</big>''':<br />  
'''<big>Development Team</big>''':<br />  
 
Product Manager: [mailto:pdolanjski@mozilla.com Peter Dolanjski] (irc: pdol)<br />
Project Manager: [mailto:elancaster@mozilla.com Erin Lancaster] (irc: elan)<br />
Engineering Manager: [mailto:pastithas@mozilla.com Panagiotis Astithas] (irc: past)<br />
UX Lead: [mailto:psackl@mozilla.com Philipp Sackl] (irc: phlsa)<br />


'''<big>Engineering Team</big> - Nightly Testing''':<br />  
'''<big>Engineering Team</big> - Nightly Testing''':<br />  
Line 37: Line 42:
[mailto:hani.yacoub@softvision.ro Hani Yacoub] (irc: hani) - QA <br />
[mailto:hani.yacoub@softvision.ro Hani Yacoub] (irc: hani) - QA <br />


= Testing summary =  
= Testing summary =  
== Scope of Testing ==
== Scope of Testing ==
Line 43: Line 47:
This feature purpose is to convert permission notifications to use two buttons instead of the drop-down menu.
This feature purpose is to convert permission notifications to use two buttons instead of the drop-down menu.


Testing will be done on 4 OSes:
The testing effort for the improve permission prompt UI will be invested on the following areas:
* Windows 10, Windows 7
 
* Mac OS X 10.11
* functionality: basic and advanced functionality to be verified according to the existing requirements;
* Ubuntu 16.04
* usability: the Indicator for device permissions must be easy to use and straightforward;
* accessibility: the Indicator for device permissions can be used with High Contrast Theme, Keyboard only or Screen Reader;


In the scope of current testing are following permissions:


=== Out of Scope ===
* Camera
Detail what is out of scope from a testing perspective for the project team. Note: if usability testing is not in the scope of testing feature.
* Microphone
* Screen/Window Sharing
* Desktop Notifications
* Geolocation
* Offilne Storage
* Pop-up Notifications
* Combination between the ones mentioned above


= Requirements for testing =
= Requirements for testing =
== Environments ==
== Environments ==
Full testing will be performed on the following OSes:
Full testing will be performed on the following OSes:
* Windows 7 and Windows 10
* Windows 10 x64, Windows 7 x32
* Mac OS X 10.12
* Mac OS X 10.10, Mac OS X 10.11 and Mac OS X 10.12
* Ubuntu 16.04
* Ubuntu 16.04


== Channel dependent settings (configs) and environment setups ==
== Channel dependent settings (configs) and environment setups ==
Line 65: Line 76:
=== Nightly ===
=== Nightly ===
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
text
</div>
</div>


=== Aurora ===
=== Aurora ===
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
text
</div>
</div>


=== Beta ===
=== Beta ===
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
text
</div>
</div>


=== Post Beta / Release ===
=== Post Beta / Release ===
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
text
</div>
</div>
</div>
</div>
Line 93: Line 100:
! Ref !! Function !! Test Objective !! Test Type !! Risk Assessment and Coverage(RAC) !! Owners  
! Ref !! Function !! Test Objective !! Test Type !! Risk Assessment and Coverage(RAC) !! Owners  
|-
|-
| 1 || Improve permission prompt UI display || The new visual design is displayed and the notification functionality is correct || Manual || RAC- || Eng Team
| TO-1 || Permission prompt for Camera || The new visual design is displayed and the notification functionality is correct || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-2 || Permission prompt for Camera functionality || The basic functions(Allowed, Allowed temporarily, Blocked and Blocked temporarily) work properly || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-3 || Permission prompt for Microphone|| The new visual design is displayed and the notification functionality is correct || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-4 || Permission prompt for Microphone functionality || The basic functions(Allowed, Allowed temporarily, Blocked and Blocked temporarily) work properly || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-5 || Permission prompt for Pointer lock|| The new visual design is displayed and the notification functionality is correct || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-6 || Permission prompt for Pointer lock functionality || The basic functions(Allowed, Allowed temporarily, Blocked and Blocked temporarily) work properly || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-7 || Permission prompt for Notifications|| The new visual design is displayed and the notification functionality is correct || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-8 || Permission prompt for Notification functionality|| The basic functions(Allowed, Allowed temporarily, Blocked and Blocked temporarily) work properly || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-9 || Permission prompt for Location|| The new visual design is displayed and the notification functionality is correct || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-10 || Permission prompt for Location functionality|| The basic functions(Allowed, Allowed temporarily, Blocked and Blocked temporarily) work properly || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-11 || Permission prompt for Popups|| The new visual design is displayed and the notification functionality is correct || Manual || RAC-1, RAC-2 || Eng Team
|-
| TO-12 || Permission prompt for Popups functionality|| The basic functions(Allowed, Blocked and Show x blocked pop-up) work properly || Manual || RAC-1, RAC-2 || Eng Team
|-
|-
| 2 || Improve permission prompt UI functionality || The basic functions(Save-don't save, Share-don't share, Allow-don't allow and always remember my decision) all work properly || Manual || RAC- || Eng Team
| TO-13 || Clearing Permissions || Validate that permission prompt set can be cleared correctly || Manual || RAC-1, RAC-2, RAC-3 || Eng Team
|-
|-
| 3 || Different permission prompt types || Validate that new permission prompt design works correctly on different permission prompt types || Manual || RAC- || Eng Team
| TO-14 || Combinations of different permission prompt types || Validate that new permission prompt design works correctly for multiple permission prompt || Manual || RAC-1, RAC-2, RAC-5 || Eng Team
|-
|-
| 4 || Performance is acceptable in case of multiple tabs and doesn't crash the browser || The objective the test is trying to demonstrate that new permission prompt design works correctly when on multiple tabs or windows || Manual || RAC- || Eng Team
| TO-15 || Performance is acceptable in case of multiple tabs and doesn't crash the browser || The objective the test is trying to demonstrate that new permission prompt design works correctly when on multiple tabs or windows || Manual || RAC-4 || Eng Team
|-
|-
| 5 || Accessibility || Testing for accessibility includes:  
| TO-16 || Accessibility || Testing for accessibility includes:  
- testing using high Contract themes <br />
- testing using high Contract themes <br />
- Screen Reader validation<br />
- Screen Reader validation<br />
- Keyboard-only   
- Keyboard-only   
|| Manual || RAC- || Eng Team
|| Manual || RAC-3, RAC-6, RAC-7 || Eng Team
|-
| TO-17 || Localization || RTL and strings in general (contextual warning) || Manual || RAC-8 || Eng Team
|}
|}


Line 114: Line 145:
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score  
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score  
|-
|-
| RAC-1 || Mouse and keyboard actions may not be recognized || TO-1 || 2-Moderate || 1-Unlikely || 3-High || 6
| RAC-1 || Refactoring of permissions prompt UI might affect Firefox base functionality  || TO-1 -> To-14 || 3-High|| 3-Almost Certain|| 3-High || 27
|-
| RAC-2 || Basic permission functionality might get broken || TO-1 -> To-14  || 3- High|| 3-Almost Certain|| 3-High || 27
|-
| RAC-3 || Mouse and keyboard actions may not be recognized || TO-13, TO-16 || 2-Moderate || 1-Unlikely || 3-High || 6
|-
| RAC-4 || Possible performance issues with multiple tabs for different permission prompt set || TO-15 || 2-Moderate || 2-Possible || 3-High || 12
|-
|-
| RAC-2 || Some Video types are not recognized and played || TO-3 || 3- High|| 2-Possible || 3-High || 18
| RAC-5 || Possible delay for the permission prompt when accessing a site || TO-14 || 3-High || 2-Possible || 3-High || 18
|-
|-
| RAC-3 || Possible performance issues with multiple tabs || TO-4 || 2-Moderate || 2-Possible || 3-High || 12
| RAC-6 || Backround colors and colors in general are visible and readable under high contrast  || TO-16  || 3-High || 3-Almost Certain|| 2-Medium || 18
|-
|-
| RAC-4 || After visiting for the first time a tab containing a video, the video starts with a delay || TO-1 || 3-High || 3-Almost Certain || 3-High || 27
| RAC-7 || Screen reader on the warning message? || TO-16 || 1- Low || 2-Possible || 3-High || 6
|-
|-
| RAC-8 || RTL  || TO-17  || 1- Low || 2-Possible || 2-Medium || 4
|}
|}


Line 133: Line 171:


'''Impact Score Breakdown:'''  
'''Impact Score Breakdown:'''  
* An impact value of 1 or 2 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 4, 8 or 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.


== Builds ==
== Builds ==
Line 152: Line 188:
|-
|-
| Start project  
| Start project  
|style="text-align:center;" | ||  
|style="text-align:center;" | 2017.01.04 ||  
|-
|-
| Study documentation/specs received from developers
| Study documentation/specs received from developers
|style="text-align:center;" |  ||  
|style="text-align:center;" |  2017.01.05
|style="text-align:center;" | 2017.01.12
|-
|-
| QA - Test plan creation  
| QA - Test plan creation  
|style="text-align:center;" |  ||  
|style="text-align:center;" |  2017.01.06
|style="text-align:center;" | 2017.01.13
|-
|-
| QA - Test cases/Env preparation  
| QA - Test cases/Env preparation  
|style="text-align:center;" | ||  
|style="text-align:center;" | 2017.01.10 ||  
|-
|-
| QA - Nightly Testing  
| QA - Nightly Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | 2017.01.12
|style="text-align:center;" | 2017.01.23
|-
|-
| QA - Aurora Testing
| QA - Aurora Testing
|style="text-align:center;" | ||
|style="text-align:center;" | 2017.01.30
|style="text-align:center;" | 2017.03.06
|-
|-
| QA - Beta Testing  
| QA - Beta Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | 2017.03.06
|style="text-align:center;" | 2017.03.31
|-
|-
| Release Date  
| Release Date  
Line 193: Line 234:
= Status =  
= Status =  
== Overview ==
== Overview ==
  Track the dates and build number where feature was released to Nightly
  First landed in Nightly 53, and we start testing 01/12/2017, with build [http://archive.mozilla.org/pub/firefox/nightly/2017/01/2017-01-12-03-03-01-mozilla-central/ Nightly link]
  Track the dates and build number where feature was merged to Aurora
  Landed on Aurora 53, on 01/23/2017, with build [http://archive.mozilla.org/pub/firefox/nightly/2017/01/2017-01-23-08-03-55-mozilla-aurora/ Aurora link]
  Track the dates and build number where feature was merged to Release/Beta
  Landed on Beta channel, with Beta 53 on 03/06/2017, with build [http://archive.mozilla.org/pub/firefox/candidates/53.0b1-candidates/ Beta 53 - b1]


== Risk analysis ==
Identify the high-risk assumptions
Identify existing bugs on the feature with high risk
Identify if other areas are affected by the fix
= References =
= References =
* Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1282768 1282768] - Convert permission notifications to use two buttons instead of the drop-down menu
* List and links for specs
* List and links for specs
   List and links for available specs - documents, user stories, specifications
   - [https://docs.google.com/document/d/143nEfWfIvFZD2-pFqE8GD85inFjFIhE2J8QeegQufc0/edit# Control Center V2 specs]
* Meta bug
  - [https://docs.google.com/document/d/1MaooflGxc2E9dcNVvz0AwYoTqySfEQOAhOpFNDsJBLs/edit# User research for Permission Prompt Redesign and Insecure Password UI]
   
  - [https://mozilla.invisionapp.com/share/9J7RMC2G7#/screens/170569580 feature samples]
* User Story Used
- [https://bugzilla.mozilla.org/show_bug.cgi?id=1188147 1188147] - [userstory] Notification of Single Permission
- [https://bugzilla.mozilla.org/show_bug.cgi?id=1188118 1188118] - [userstory] ID Block: Control Center "i" anchor icon
- [https://bugzilla.mozilla.org/show_bug.cgi?id=1188152 1188152] - [userstory] Notification when another request is made (multiple state)
- [https://bugzilla.mozilla.org/show_bug.cgi?id=1188355 1188355] - [userstory] CC: Site Permissions on main panel
  - [https://bugzilla.mozilla.org/show_bug.cgi?id=1188472 1188472] - [userstory] ID Block: Permission icon when blocked
 
= Testcases =  
= Testcases =  
== Overview ==
Summary of testing scenarios


== Test Areas ==
== Test Areas ==
Line 217: Line 259:
|-
|-
| Private Window  
| Private Window  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Multi-Process Enabled  
| Multi-Process Enabled  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Multi-process Disabled  
| Multi-process Disabled  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Theme (high contrast)  
| Theme (high contrast)  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| '''UI'''  
| '''UI'''  
Line 232: Line 274:
|-
|-
| Mouse-only operation   
| Mouse-only operation   
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Keyboard-only operation   
| Keyboard-only operation   
|style="text-align:center;" | ||
|style="text-align:center;" | Yes ||
|-
|-
| Display (HiDPI)  
| Display (HiDPI)  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Interraction (scroll, zoom)  
| Interraction (scroll, zoom)  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Usable with a screen reader   
| Usable with a screen reader   
|style="text-align:center;" | || e.g. with NVDA
|style="text-align:center;" | Yes || e.g. with NVDA
|-
|-
| Usability and/or discoverability testing   
| Usability and/or discoverability testing   
|style="text-align:center;" | || Is this feature user friendly
|style="text-align:center;" | Yes || Is this feature user friendly
|-
|-
| RTL build testing   
| RTL build testing   
|style="text-align:center;" | ||
|style="text-align:center;" | No||
|-
|-
| '''Help/Support'''  
| '''Help/Support'''  
Line 256: Line 298:
|-
|-
| Help/support interface required   
| Help/support interface required   
|style="text-align:center;" | || Make sure link to support/help page exist and is easy reachable.
|style="text-align:center;" | No || Make sure link to support/help page exist and is easy reachable.
|-
|-
| Support documents planned(written)   
| Support documents planned(written)   
|style="text-align:center;" | || Make sure support documents are written and are correct.
|style="text-align:center;" | No || Make sure support documents are written and are correct.


|-
|-
Line 266: Line 308:
|-
|-
| Feature upgrades/downgrades data as expected   
| Feature upgrades/downgrades data as expected   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||
|-
|-
| Does sync work across upgrades   
| Does sync work across upgrades   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Requires install testing     
| Requires install testing     
|style="text-align:center;" | || separate feature/application installation needed (not only Firefox)
|style="text-align:center;" | No || separate feature/application installation needed (not only Firefox)
|-
|-
| Affects first-run or onboarding     
| Affects first-run or onboarding     
|style="text-align:center;" | || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|style="text-align:center;" | N/A || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|-
|-
| Does this affect partner builds? Partner build testing   
| Does this affect partner builds? Partner build testing   
|style="text-align:center;" | || yes/no options, add comment with details about who will lead testing
|style="text-align:center;" | N/A || yes/no options, add comment with details about who will lead testing


|-
|-
Line 285: Line 327:
|-
|-
| Enterprise administration   
| Enterprise administration   
|style="text-align:center;" | ||  
|style="text-align:center;" | N/A ||  
|-
|-
| Network proxies/autoconfig   
| Network proxies/autoconfig   
|style="text-align:center;" | ||  
|style="text-align:center;" | N/A ||  
|-
|-
| ESR behavior changes   
| ESR behavior changes   
|style="text-align:center;" | ||  
|style="text-align:center;" | N/A ||  
|-
|-
| Locked preferences   
| Locked preferences   
|style="text-align:center;" | ||
|style="text-align:center;" | N/A ||


|-
|-
Line 301: Line 343:
|-
|-
| Temporary or permanent telemetry monitoring   
| Temporary or permanent telemetry monitoring   
|style="text-align:center;" | || List of error conditions to monitor
|style="text-align:center;" | N/A || List of error conditions to monitor
|-
|-
| Telemetry correctness testing   
| Telemetry correctness testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | N/A ||  
|-
|-
| Server integration testing   
| Server integration testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Offline and server failure testing   
| Offline and server failure testing   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||
|-
|-
| Load testing   
| Load testing   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||


|-
|-
Line 320: Line 362:
|-
|-
| Addon API required?   
| Addon API required?   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Comprehensive API testing   
| Comprehensive API testing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Permissions   
| Permissions   
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Testing with existing/popular addons
| Testing with existing/popular addons
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  


|-
|-
Line 336: Line 378:
|-
|-
| 3rd-party security review   
| 3rd-party security review   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Privilege escalation testing
| Privilege escalation testing
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Fuzzing   
| Fuzzing   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  


|-
|-
Line 349: Line 391:
|-
|-
| Testing against target sites   
| Testing against target sites   
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  
|-
|-
| Survey of many sites for compatibility   
| Survey of many sites for compatibility   
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes ||  


|-
|-
Line 359: Line 401:
|-
|-
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|-
|-
| Interaction of this feature with other browser features   
| Interaction of this feature with other browser features   
|style="text-align:center;" | ||  
|style="text-align:center;" | No ||  
|}
|}


== Test suite ==
== Test suite ==
  Full Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/1N8dGK_uenhfcM9cEPC3HPHmi0jFN98FWtL2DuszrfLY/edit#gid=0 link]
  Full Test suite - in TestRail at [https://testrail.stage.mozaws.net/index.php?/suites/view/423&group_by=cases:section_id&group_order=asc link]
Smoke Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/1N8dGK_uenhfcM9cEPC3HPHmi0jFN98FWtL2DuszrfLY/edit#gid=0 link]
Regression Test suite - Link with the gdoc - if available/needed.


= Bug Work =
= Bug Work =
Line 404: Line 444:
Check list
Check list
* All test cases should be executed
* All test cases should be executed
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)


Line 410: Line 449:
'''Nightly testing'''<br />
'''Nightly testing'''<br />


List of OSes that will be covered by testing<br />
*List of OSes that will be covered by testing<br />
** Windows 7 x64
** Ubuntu 16.04 x64
** Mac OS X 10.11 (El Capitan)
*Link for the tests run
*Link for the tests run
** Daily Smoke, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
** Full Test suite - [https://testrail.stage.mozaws.net/index.php?/runs/view/1949&group_by=cases:section_id&group_order=asc Nightly tests]
** Full Test suite, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
*This feature was treated as bug work, so no sign-off email was sent
** Regression Test suite, if needed/available


'''Merge to Aurora Sign-off'''
'''Merge to Aurora Sign-off'''<br />
List of OSes that will be covered by testing<br />
*List of OSes that will be covered by testing<br />
** Windows 10 x64
** Ubuntu 16.04 x64
** Mac OS X 10.12 (Sierra)
*Link for the tests run
*Link for the tests run
** Full Test suite
** Full Test suite - [https://testrail.stage.mozaws.net/index.php?/runs/view/2470&group_by=cases:section_id&group_order=asc Aurora tests]
*This feature was treated as bug work, so no sign-off email was sent
 
'''Merge to Beta/Pre-Release Sign-off'''
*List of OSes that will be covered by testing<br />
** Windows 10 x64
** Windows 7 x32
** Ubuntu 16.04 x64
** Mac OS X 10.12 (Sierra)
*Link for the tests run
** Full Test suite - [https://testrail.stage.mozaws.net/index.php?/runs/view/3335&group_by=cases:section_id&group_order=asc Beta test runs]


== Checklist ==
== Checklist ==
Line 427: Line 481:
|-
|-
|  Testing Prerequisites (specs, use cases)  
|  Testing Prerequisites (specs, use cases)  
| style="text-align:center;" |   
| style="text-align:center;" |  {{mdone|}}
| style="text-align:center;" |  
| style="text-align:center;" |  
|-
|-
|  Testing Infrastructure setup  
|  Testing Infrastructure setup  
|style="text-align:center;" |   ||  
|style="text-align:center;" | {{mdone|}}  ||  
|-
|-
|  Test Plan Creation  
|  Test Plan Creation  
| style="text-align:center;" |   ||  
| style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  Test Cases Creation  
|  Test Cases Creation  
|style="text-align:center;" |   ||  
|style="text-align:center;" | {{mdone|}}  ||  
|-
|-
|  Full Functional Tests Execution  
|  Full Functional Tests Execution  
|style="text-align:center;" |   
|style="text-align:center;" |  {{mdone|}}
|style="text-align:center;" |   
|style="text-align:center;" |   
|-
|-
|  Automation Coverage ||
|  Automation Coverage  
|style="text-align:center;" |  
|style="text-align:center;" | {{mdone|}}
|style="text-align:center;" |
|-
|-
|  Performance Testing  
|  Performance Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  All Defects Logged || ||  
|  All Defects Logged  
|style="text-align:center;" | {{mdone|}}||  
|-
|-
|  Critical/Blockers Fixed and Verified || ||  
|  Critical/Blockers Fixed and Verified  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  Metrics/Telemetry||  
|  Metrics/Telemetry||  
|style="text-align:center;" |  
|style="text-align:center;" |  
|-
|-
|  QA Signoff  - Nightly Release||  
|  QA Signoff  - Nightly Release
|style="text-align:center;" | Email to be sent  
|style="text-align:center;" | N/A
|style="text-align:center;" | Was treated as bug work - no sign off email was sent.
|-
|-
|  QA Aurora - Full Testing  
|  QA Aurora - Full Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  QA Signoff  - Aurora Release ||  
|  QA Signoff  - Aurora Release  
|style="text-align:center;"| Email to be sent  
|style="text-align:center;" | N/A
|style="text-align:center;"| Was treated as bug work - no sign off email was sent.
|-
|-
|  QA Beta - Full Testing
|  QA Beta - Full Testing
|style="text-align:center;" | ||  
|style="text-align:center;" | {{mdone|}} ||  
|-
|-
|  QA Signoff  - Beta Release ||  
|  QA Signoff  - Beta Release ||  
|style="text-align:center;" | Email to be sent  
|style="text-align:center;" | Email to be sent  
|}
|}

Latest revision as of 06:51, 6 April 2017

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
01/06/2017 1.0 Hani Yacoub Created first draft

Overview

Purpose

Detail the purpose of this document. For example:

  • The test scope, focus areas and objectives
  • The test responsibilities
  • The test strategy for the levels and types of test for this release
  • The entry and exit criteria
  • The basis of the test estimates
  • Any risks, issues, assumptions and test dependencies
  • The test schedule and major milestones
  • The test deliverables

Scope

This wiki details the testing that will be performed by the project team for the Permissions Prompt Improvements project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document what and how we shall test:

  • Permissions are listed and correctly managed for devices actively streaming
  • The area involving includes every permission doorhanger used in Firefox, with the exception of the plugin prompt.
  • Behavior of the permission prompt when the resources (audio/video/sharing) are called from a sub-frame
  • Verify permission prompt on secure and unsecured sites.

Ownership

Development Team:
Product Manager: Peter Dolanjski (irc: pdol)
Project Manager: Erin Lancaster (irc: elan)
Engineering Manager: Panagiotis Astithas (irc: past)
UX Lead: Philipp Sackl (irc: phlsa)

Engineering Team - Nightly Testing:
Rares Bologa (irc: RaresB) - PM for QA team
Brindusa Tot (irc: brindusat) - leading QA efforts
Hani Yacoub (irc: hani) - QA

Testing summary

Scope of Testing

In Scope

This feature purpose is to convert permission notifications to use two buttons instead of the drop-down menu.

The testing effort for the improve permission prompt UI will be invested on the following areas:

  • functionality: basic and advanced functionality to be verified according to the existing requirements;
  • usability: the Indicator for device permissions must be easy to use and straightforward;
  • accessibility: the Indicator for device permissions can be used with High Contrast Theme, Keyboard only or Screen Reader;

In the scope of current testing are following permissions:

  • Camera
  • Microphone
  • Screen/Window Sharing
  • Desktop Notifications
  • Geolocation
  • Offilne Storage
  • Pop-up Notifications
  • Combination between the ones mentioned above

Requirements for testing

Environments

Full testing will be performed on the following OSes:

  • Windows 10 x64, Windows 7 x32
  • Mac OS X 10.10, Mac OS X 10.11 and Mac OS X 10.12
  • Ubuntu 16.04

Channel dependent settings (configs) and environment setups

Nightly

Aurora

Beta

Post Beta / Release

Test Strategy

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref Function Test Objective Test Type Risk Assessment and Coverage(RAC) Owners
TO-1 Permission prompt for Camera The new visual design is displayed and the notification functionality is correct Manual RAC-1, RAC-2 Eng Team
TO-2 Permission prompt for Camera functionality The basic functions(Allowed, Allowed temporarily, Blocked and Blocked temporarily) work properly Manual RAC-1, RAC-2 Eng Team
TO-3 Permission prompt for Microphone The new visual design is displayed and the notification functionality is correct Manual RAC-1, RAC-2 Eng Team
TO-4 Permission prompt for Microphone functionality The basic functions(Allowed, Allowed temporarily, Blocked and Blocked temporarily) work properly Manual RAC-1, RAC-2 Eng Team
TO-5 Permission prompt for Pointer lock The new visual design is displayed and the notification functionality is correct Manual RAC-1, RAC-2 Eng Team
TO-6 Permission prompt for Pointer lock functionality The basic functions(Allowed, Allowed temporarily, Blocked and Blocked temporarily) work properly Manual RAC-1, RAC-2 Eng Team
TO-7 Permission prompt for Notifications The new visual design is displayed and the notification functionality is correct Manual RAC-1, RAC-2 Eng Team
TO-8 Permission prompt for Notification functionality The basic functions(Allowed, Allowed temporarily, Blocked and Blocked temporarily) work properly Manual RAC-1, RAC-2 Eng Team
TO-9 Permission prompt for Location The new visual design is displayed and the notification functionality is correct Manual RAC-1, RAC-2 Eng Team
TO-10 Permission prompt for Location functionality The basic functions(Allowed, Allowed temporarily, Blocked and Blocked temporarily) work properly Manual RAC-1, RAC-2 Eng Team
TO-11 Permission prompt for Popups The new visual design is displayed and the notification functionality is correct Manual RAC-1, RAC-2 Eng Team
TO-12 Permission prompt for Popups functionality The basic functions(Allowed, Blocked and Show x blocked pop-up) work properly Manual RAC-1, RAC-2 Eng Team
TO-13 Clearing Permissions Validate that permission prompt set can be cleared correctly Manual RAC-1, RAC-2, RAC-3 Eng Team
TO-14 Combinations of different permission prompt types Validate that new permission prompt design works correctly for multiple permission prompt Manual RAC-1, RAC-2, RAC-5 Eng Team
TO-15 Performance is acceptable in case of multiple tabs and doesn't crash the browser The objective the test is trying to demonstrate that new permission prompt design works correctly when on multiple tabs or windows Manual RAC-4 Eng Team
TO-16 Accessibility Testing for accessibility includes:

- testing using high Contract themes
- Screen Reader validation
- Keyboard-only

Manual RAC-3, RAC-6, RAC-7 Eng Team
TO-17 Localization RTL and strings in general (contextual warning) Manual RAC-8 Eng Team

Risk Assessment and Coverage

ID Description / Threat Description Covered by Test Objective Magnitude Probability Priority Impact Score
RAC-1 Refactoring of permissions prompt UI might affect Firefox base functionality TO-1 -> To-14 3-High 3-Almost Certain 3-High 27
RAC-2 Basic permission functionality might get broken TO-1 -> To-14 3- High 3-Almost Certain 3-High 27
RAC-3 Mouse and keyboard actions may not be recognized TO-13, TO-16 2-Moderate 1-Unlikely 3-High 6
RAC-4 Possible performance issues with multiple tabs for different permission prompt set TO-15 2-Moderate 2-Possible 3-High 12
RAC-5 Possible delay for the permission prompt when accessing a site TO-14 3-High 2-Possible 3-High 18
RAC-6 Backround colors and colors in general are visible and readable under high contrast TO-16 3-High 3-Almost Certain 2-Medium 18
RAC-7 Screen reader on the warning message? TO-16 1- Low 2-Possible 3-High 6
RAC-8 RTL TO-17 1- Low 2-Possible 2-Medium 4

Values:

  • Magnitude: 1- Low , 2-Moderate, 3-High
  • Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
  • Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

  • An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
  • An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
  • An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

Builds

This section should contain links for builds with the feature -

  • Links for Nightly builds
  • Links for Aurora builds
  • Links for Beta builds

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 2017.01.04
Study documentation/specs received from developers 2017.01.05 2017.01.12
QA - Test plan creation 2017.01.06 2017.01.13
QA - Test cases/Env preparation 2017.01.10
QA - Nightly Testing 2017.01.12 2017.01.23
QA - Aurora Testing 2017.01.30 2017.03.06
QA - Beta Testing 2017.03.06 2017.03.31
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail/ Google docs
Test case execution TestRail
Bugs management Bugzilla

Status

Overview

First landed in Nightly 53, and we start testing 01/12/2017, with build Nightly link
Landed on Aurora 53, on 01/23/2017, with build Aurora link
Landed on Beta channel, with Beta 53 on 03/06/2017, with build Beta 53 - b1

References

  • Meta bug 1282768 - Convert permission notifications to use two buttons instead of the drop-down menu
  • List and links for specs
 - Control Center V2 specs
 - User research for Permission Prompt Redesign and Insecure Password UI
 - feature samples
  • User Story Used
- 1188147 - [userstory] Notification of Single Permission
- 1188118 - [userstory] ID Block: Control Center "i" anchor icon
- 1188152 - [userstory] Notification when another request is made (multiple state)
- 1188355 - [userstory] CC: Site Permissions on main panel
- 1188472 - [userstory] ID Block: Permission icon when blocked

Testcases

Test Areas

Test Areas Covered Details
Private Window Yes
Multi-Process Enabled Yes
Multi-process Disabled Yes
Theme (high contrast) Yes
UI
Mouse-only operation Yes
Keyboard-only operation Yes
Display (HiDPI) Yes
Interraction (scroll, zoom) Yes
Usable with a screen reader Yes e.g. with NVDA
Usability and/or discoverability testing Yes Is this feature user friendly
RTL build testing No
Help/Support
Help/support interface required No Make sure link to support/help page exist and is easy reachable.
Support documents planned(written) No Make sure support documents are written and are correct.
Install/Upgrade
Feature upgrades/downgrades data as expected No
Does sync work across upgrades No
Requires install testing No separate feature/application installation needed (not only Firefox)
Affects first-run or onboarding N/A Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
Does this affect partner builds? Partner build testing N/A yes/no options, add comment with details about who will lead testing
Enterprise Raise up the topic to developers to see if they are expecting to work different on ESR builds
Enterprise administration N/A
Network proxies/autoconfig N/A
ESR behavior changes N/A
Locked preferences N/A
Data Monitoring
Temporary or permanent telemetry monitoring N/A List of error conditions to monitor
Telemetry correctness testing N/A
Server integration testing No
Offline and server failure testing No
Load testing No
Add-ons If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required? No
Comprehensive API testing No
Permissions Yes
Testing with existing/popular addons No
Security Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
3rd-party security review No
Privilege escalation testing No
Fuzzing No
Web Compatibility depends on the feature
Testing against target sites Yes
Survey of many sites for compatibility Yes
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. No
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS No
Interaction of this feature with other browser features No

Test suite

Full Test suite - in TestRail at link

Bug Work

Tracking bug – meta bug

Bug fix verification
Bug No Summary Status Firefox Verion
123 bug summary NEW Nighly 45
Logged bugs

Bug 111111
Bug 211111


Sign off

Criteria

Check list

  • All test cases should be executed
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

  • List of OSes that will be covered by testing
    • Windows 7 x64
    • Ubuntu 16.04 x64
    • Mac OS X 10.11 (El Capitan)
  • Link for the tests run
  • This feature was treated as bug work, so no sign-off email was sent

Merge to Aurora Sign-off

  • List of OSes that will be covered by testing
    • Windows 10 x64
    • Ubuntu 16.04 x64
    • Mac OS X 10.12 (Sierra)
  • Link for the tests run
  • This feature was treated as bug work, so no sign-off email was sent

Merge to Beta/Pre-Release Sign-off

  • List of OSes that will be covered by testing
    • Windows 10 x64
    • Windows 7 x32
    • Ubuntu 16.04 x64
    • Mac OS X 10.12 (Sierra)
  • Link for the tests run

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [DONE]
Testing Infrastructure setup [DONE]
Test Plan Creation [DONE]
Test Cases Creation [DONE]
Full Functional Tests Execution [DONE]
Automation Coverage [DONE]
Performance Testing [DONE]
All Defects Logged [DONE]
Critical/Blockers Fixed and Verified [DONE]
Metrics/Telemetry
QA Signoff - Nightly Release N/A Was treated as bug work - no sign off email was sent.
QA Aurora - Full Testing [DONE]
QA Signoff - Aurora Release N/A Was treated as bug work - no sign off email was sent.
QA Beta - Full Testing [DONE]
QA Signoff - Beta Release Email to be sent
Retrieved from "https://wiki.allizom.org/index.php?title=QA/improve_permission_prompt_UI&oldid=1167677"