QA/Flash SHIELD Experiment: Difference between revisions

From MozillaWiki
< QA
Jump to navigation Jump to search
No edit summary
 
(26 intermediate revisions by 3 users not shown)
Line 8: Line 8:
|-
|-
| 12/08/2016 || 1.0 || Stefan Georgiev || Created first draft
| 12/08/2016 || 1.0 || Stefan Georgiev || Created first draft
|-
| 02/16/2018 || 1.1 || Stefan Georgiev || Updated first draft
|-
|-
|}
|}
Line 19: Line 21:


* Goals
* Goals
Block Flash by default
Block Flash by default<br />
Keep users secure from Flash exploits
Keep users secure from Flash exploits
* How testing will be performed
* How testing will be performed
Line 25: Line 27:


== Ownership ==
== Ownership ==
'''Mozilla Development'''<br />


'''QA Eng. Softvision Las Vegas'''<br />
Product Manager: Benjamin Smedberg
[mailto:stefan.georgiev@softvision.com Stefan Georgiev - Team Lead]<br />
Engineering Lead: Felipe Gomes
[mailto:gwimberly@softvision.com Grover Wimberly IV - Tester]<br />
Engineering Manager: David Durst
[mailto:kkumari@softvision.com Kanchan Kumari - Tester]<br />
Test Lead: [mailto:stefan.georgiev@softvision.com Stefan Georgiev]
[mailto:amasresha@softvision.com Abe Masresha - Tester]<br />
[mailto:jwilliams@softvision.com Justin Williams - Tester]<br />


= Testing summary =  
= Testing summary =  
Line 41: Line 39:


=== Out of Scope ===
=== Out of Scope ===
Android & iOS Devices
Android & iOS Devices<br />
Exclude users without Flash<br />
Exclude users who don’t have the targeted gecko version


= Requirements for testing =
= Requirements for testing =
Line 54: Line 54:
== Channel dependent settings (configs) and environment setups ==
== Channel dependent settings (configs) and environment setups ==
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">


=== Post Beta / Release ===
=== Post Beta / Release ===
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
Not Applicable
</div>
</div>
</div>
</div>
Line 64: Line 62:
= Test Strategy =  
= Test Strategy =  
== Test Objectives ==
== Test Objectives ==
Verify Camera and Microphone<br />
URLs<br />
Verify Screen Sharing (Single/Dual Monitors) [Cisco Spark]<br />
Lists<br />
Verify Application Sharing<br />
Heuristics<br />
Verify Window Sharing<br />


== Builds ==
== Builds ==


* Links for Nightly builds:  
* Links for Test builds:  
#Latest Mozilla Central Nightly Builds [[http://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/]]
#Latest Mozilla Test Builds [[]]
 


== Test Execution Schedule ==
== Test Execution Schedule ==
Line 85: Line 81:
|-
|-
| Study documentation/specs received from developers
| Study documentation/specs received from developers
|style="text-align:center;" |10/24/2016  ||  
|style="text-align:center;" |12/06/2016  ||  
|-
|-
| QA - Test plan creation  
| QA - Test plan creation  
|style="text-align:center;" | 10/27/2016 ||  
|style="text-align:center;" | 12/08/2016 ||  
|-
|-
| QA - Test cases/Env preparation  
| QA - Test cases/Env preparation  
|style="text-align:center;" | 10/25/2016 ||  
|style="text-align:center;" |  ||  
|-
|-
| QA - Nightly Testing  
| QA - Nightly Testing  
Line 127: Line 123:


== Risk analysis ==
== Risk analysis ==
Accidental exposure -> sharing information on the desktop not intended to share
* Breaking websites:
** Top destinations websites: Facebook, Yahoo, Google, Gmail, Google Docs, etc.
** Top media websites: Youtube, Vimeo, Spotify, some news websites with video
** Top Flash-based game websites: Kongregate, etc.
 
* Verify that no site gets a broken or "transparent" ad on top of content that never loads
 
* Verify that no site gets in a forever-loading state due to this change
 
* Verify that no top-site hangs or significantly slows down Firefox due to this change
 
* Verify that no data from the experiment is missing or incorrect
 
* Check that heuristics are correctly applied
 
* Check that whitelist/blocklist works correctly
 
= References =
= References =
#Screen Sharing [[https://docs.google.com/document/d/1m1arSi9mr2qrOuzP9B-Oz1qG51Sy1xfPjBpHekpGJNc/edit?pref=2&pli=1]]
* List and links for specs - Specs [https://docs.google.com/document/d/1sYp0DNioPA5iF3iw9LHGf1uN5B5AgdCu7jJxAh0MiqA/edit#]
#Show Preview Bug 1284877#c0 [[https://bugzilla.mozilla.org/show_bug.cgi?id=1284877#c0]]
 
#Mock up [[https://mozilla.invisionapp.com/share/AF71R266U#/screens/152725629]]
* Meta bug - Bug 1335232 [https://bugzilla.mozilla.org/show_bug.cgi?id=1335232 ]


* Meta bug
= Testcases =  
= Testcases =  
== Overview ==
== Overview ==
  Summary of testing scenarios
  Summary of testing scenarios
1. URL
*Verify allowed list is applied - ''Flash is included in the navigator.plugins, Flash instances are activated immediately''
*Verify blocked list is applied - ''Flash is not included in the navigator.plugins, any attempt to use Flash use fallback content''
*Verify for all essential Flash, fallback (non-Flash) format is used to shown the content
*Verify Flash content is set to Click-to-active
*Verify allowed/denied list is set to geography-specific
*User is able to enable the Flash content
*User is able to block the Flash content
*Browsers send correct telemetry data to the server
*User has explicitly set "Disallow Flash" for this site - ''Flash content should be "click to activate" / Flash is present in the navigator.plugins''
*User has explicitly set to "Allow Flash" for this site - ''Flash is present in the navigator.plugins / Flash elements are automatically activated''
2. Lists
*Check whitelisted domain - ''page loads; Flash is not set to CTA - check iframes against the list''
*Check blacklisted domain - ''page loads; Flash is not loaded - don't check iframes against the list''
*Check non-whitelisted/non-blacklisted domain page loads - ''check all iframes against the list''
3. Heuristics
*Flash has acceptable fallback content - ''Show the fallback content''
*Flash doesn't have acceptable fallback content
** ''If it pass the heuristics - set Flash to CTA''
** ''If it doesn't pass the heuristics - don't load the Flash content''


== Test Areas ==
== Test Areas ==
Line 152: Line 199:
|-
|-
| Multi-process Disabled  
| Multi-process Disabled  
|style="text-align:center;" |  || Only tested if there are bugs found in testing
|style="text-align:center;" |  || To be tested
|-
|-
| Theme (high contrast)  
| Theme (high contrast)  
Line 167: Line 214:
|-
|-
| Display (HiDPI)  
| Display (HiDPI)  
|style="text-align:center;" | || To be tested
|style="text-align:center;" | || Not applicable
|-
|-
| Interraction (scroll, zoom)  
| Interraction (scroll, zoom)  
|style="text-align:center;" | || To be tested
|style="text-align:center;" | || Not applicable
|-
|-
| Usable with a screen reader   
| Usable with a screen reader   
|style="text-align:center;" |  || e.g. with NVDA
|style="text-align:center;" |  || Not applicable
|-
|-
| Usability and/or discoverability testing   
| Usability and/or discoverability testing   
Line 179: Line 226:
|-
|-
| RTL build testing   
| RTL build testing   
|style="text-align:center;" |  || To be tested
|style="text-align:center;" |  || Not applicable
|-
|-
| '''Help/Support'''  
| '''Help/Support'''  
Line 189: Line 236:
| Support documents planned(written)   
| Support documents planned(written)   
|style="text-align:center;" |  || Make sure support documents are written and are correct.
|style="text-align:center;" |  || Make sure support documents are written and are correct.
|-
|-
| '''Install/Upgrade'''  
| '''Install/Upgrade'''  
Line 201: Line 247:
|-
|-
| Requires install testing     
| Requires install testing     
|style="text-align:center;" |  || Default installation with Nightly
|style="text-align:center;" |  || Default installation with Test Build
|-
|-
| Affects first-run or onboarding     
| Affects first-run or onboarding     
|style="text-align:center;" |  || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|style="text-align:center;" |  || To be tested
|-
|-
| Does this affect partner builds? Partner build testing 
|style="text-align:center;" |  || yes/no options, add comment with details about who will lead testing


|-
|-
| ''' Enterprise '''  
| ''' Enterprise '''  
||  || Raise up the topic to developers to see if they are expecting to work different on ESR builds
||  ||
|-
|-
| Enterprise administration   
| Enterprise administration   
Line 233: Line 277:
|-
|-
| Telemetry correctness testing   
| Telemetry correctness testing   
|style="text-align:center;" |  || Not applicable
|style="text-align:center;" |  || To be tested
|-
|-
| Server integration testing   
| Server integration testing   
|style="text-align:center;" |  || Not applicable
|style="text-align:center;" |  || To be tested
|-
|-
| Offline and server failure testing   
| Offline and server failure testing   
|style="text-align:center;" |  || Not applicable
|style="text-align:center;" |  || To be tested
|-
|-
| Load testing   
| Load testing   
|style="text-align:center;" |  || Not applicable
|style="text-align:center;" |  || Not applicable
|-
|-
| ''' Add-ons '''  
| ''' Add-ons '''  
||  || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
||  ||  
|-
|-
| Addon API required?   
| Addon API required?   
Line 255: Line 298:
|-
|-
| Permissions   
| Permissions   
|style="text-align:center;" |  || Permission manager for sharing microphone, camera, and screen/monitor.
|style="text-align:center;" |  || Not applicable
|-
|-
| Testing with existing/popular addons
| Testing with existing/popular addons
|style="text-align:center;" |  || Not applicable
|style="text-align:center;" |  || To be tested
 
|-
|-
| ''' Security '''  
| ''' Security '''  
||  || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
||  ||  
|-
|-
| 3rd-party security review   
| 3rd-party security review   
Line 272: Line 314:
| Fuzzing   
| Fuzzing   
|style="text-align:center;" |  || Not applicable
|style="text-align:center;" |  || Not applicable
|-
|-
| ''' Web Compatibility '''  
| ''' Web Compatibility '''  
||  || depends on the feature
||  ||  
|-
|-
| Testing against target sites   
| Testing against target sites   
|style="text-align:center;" |  || CiscoSpark, Skype, GoToMeeting, free web communication services
|style="text-align:center;" |  || To be tested
|-
|-
| Survey of many sites for compatibility   
| Survey of many sites for compatibility   
|style="text-align:center;" |  || Not applicable
|style="text-align:center;" |  || To be tested
 
|-
|-
| ''' Interoperability  '''  
| ''' Interoperability  '''  
||  || depends on the feature
||  ||  
|-
|-
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
|style="text-align:center;" |  || Some tests require 2+ people to be able to do
|style="text-align:center;" |  ||  
|-
|-
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
Line 298: Line 338:


== Test suite ==
== Test suite ==
  Full Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/1N8dGK_uenhfcM9cEPC3HPHmi0jFN98FWtL2DuszrfLY/edit#gid=0 link]
  Full Test suite - [TestRail]
Smoke Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/1N8dGK_uenhfcM9cEPC3HPHmi0jFN98FWtL2DuszrfLY/edit#gid=0 link]
Regression Test suite - Link with the gdoc - if available/needed.


= Bug Work =
= Bug Work =
Tracking bug – meta bug
Tracking bug – meta bug 1277346 [https://bugzilla.mozilla.org/show_bug.cgi?id=1277346]<br />
Tracking bug - bug 1282484 [https://bugzilla.mozilla.org/show_bug.cgi?id=1282484]


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
Line 313: Line 352:
! Bug No !! Summary !! Status !! Firefox Verion
! Bug No !! Summary !! Status !! Firefox Verion
|-
|-
|  [TBA] || bug summary || NEW || Nightly
|  [https://bugzilla.mozilla.org/show_bug.cgi?id= ] Empty || ||Build  ||
|-
|-
| || || ||  
| [https://bugzilla.mozilla.org/show_bug.cgi?id= ] Empty || ||Build  ||  
|}
|}
</div></div>
</div></div>
Line 323: Line 362:
====== Logged bugs ======
====== Logged bugs ======
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
[https://bugzilla.mozilla.org/show_bug.cgi?id=1284877 Bug 1284877]<br/>
 
<br/>
</div>
</div>
</div>
</div>
Line 332: Line 370:
Check list
Check list
* All test cases should be executed
* All test cases should be executed
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/QA)
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)


== Results ==
== Results ==
'''Nightly testing'''<br />
List of OSes that will be covered by testing<br />
*Link for the tests run
*Link for the tests run
** Daily Smoke, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
** Daily Smoke, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
** Full Test suite, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
** Full Test suite, use template from [https://docs.google.com/spreadsheets/d/1ch0OubyOWn1W-_JLF-38nvSvVB76Wmo1hkLccEtU5hk/edit?usp=sharing link]
** Regression Test suite, if needed/available
** Regression Test suite, if needed/available
'''Merge to Aurora Sign-off'''
List of OSes that will be covered by testing<br />
*Link for the tests run
** Full Test suite


== Checklist ==
== Checklist ==

Latest revision as of 21:24, 21 February 2017

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
12/08/2016 1.0 Stefan Georgiev Created first draft
02/16/2018 1.1 Stefan Georgiev Updated first draft

Overview

Purpose

Most of the browsers are planning to move to click-to-activate (CTA). But before Firefox prepare for this shift, we want to understand the impact on the user and the approach we should take.

Scope

This wiki details the testing that will be performed by the project team for the Flash SHIELD Experiment project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • Goals

Block Flash by default
Keep users secure from Flash exploits

  • How testing will be performed

Manual

Ownership

Product Manager: Benjamin Smedberg Engineering Lead: Felipe Gomes Engineering Manager: David Durst Test Lead: Stefan Georgiev

Testing summary

Scope of Testing

In Scope

This feature will be tested manually on Windows, OSX and Linux operating systems.

Out of Scope

Android & iOS Devices
Exclude users without Flash
Exclude users who don’t have the targeted gecko version

Requirements for testing

Environments

Windows 7
Windows 8.1
Windows 10
Ubuntu/Linux
Macintosh 10.12


Channel dependent settings (configs) and environment setups

Post Beta / Release

Test Strategy

Test Objectives

URLs
Lists
Heuristics

Builds

  • Links for Test builds:
  1. Latest Mozilla Test Builds [[]]

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project
Study documentation/specs received from developers 12/06/2016
QA - Test plan creation 12/08/2016
QA - Test cases/Env preparation
QA - Nightly Testing
QA - Aurora Testing
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail/ Google docs
Test case execution TestRail
Bugs management Bugzilla

Status

Overview

Track the dates and build number where feature was released to Nightly
TBD

Risk analysis

  • Breaking websites:
    • Top destinations websites: Facebook, Yahoo, Google, Gmail, Google Docs, etc.
    • Top media websites: Youtube, Vimeo, Spotify, some news websites with video
    • Top Flash-based game websites: Kongregate, etc.
  • Verify that no site gets a broken or "transparent" ad on top of content that never loads
  • Verify that no site gets in a forever-loading state due to this change
  • Verify that no top-site hangs or significantly slows down Firefox due to this change
  • Verify that no data from the experiment is missing or incorrect
  • Check that heuristics are correctly applied
  • Check that whitelist/blocklist works correctly

References

  • List and links for specs - Specs [1]
  • Meta bug - Bug 1335232 [2]

Testcases

Overview

Summary of testing scenarios

1. URL

  • Verify allowed list is applied - Flash is included in the navigator.plugins, Flash instances are activated immediately
  • Verify blocked list is applied - Flash is not included in the navigator.plugins, any attempt to use Flash use fallback content
  • Verify for all essential Flash, fallback (non-Flash) format is used to shown the content
  • Verify Flash content is set to Click-to-active
  • Verify allowed/denied list is set to geography-specific
  • User is able to enable the Flash content
  • User is able to block the Flash content
  • Browsers send correct telemetry data to the server
  • User has explicitly set "Disallow Flash" for this site - Flash content should be "click to activate" / Flash is present in the navigator.plugins
  • User has explicitly set to "Allow Flash" for this site - Flash is present in the navigator.plugins / Flash elements are automatically activated

2. Lists

  • Check whitelisted domain - page loads; Flash is not set to CTA - check iframes against the list
  • Check blacklisted domain - page loads; Flash is not loaded - don't check iframes against the list
  • Check non-whitelisted/non-blacklisted domain page loads - check all iframes against the list

3. Heuristics

  • Flash has acceptable fallback content - Show the fallback content
  • Flash doesn't have acceptable fallback content
    • If it pass the heuristics - set Flash to CTA
    • If it doesn't pass the heuristics - don't load the Flash content

Test Areas

Test Areas Covered Details
Private Window Certain tests are performed in a private window.
Multi-Process Enabled Enabled by default
Multi-process Disabled To be tested
Theme (high contrast) Not applicable
UI
Mouse-only operation To be tested
Keyboard-only operation To be tested
Display (HiDPI) Not applicable
Interraction (scroll, zoom) Not applicable
Usable with a screen reader Not applicable
Usability and/or discoverability testing Is this feature user friendly
RTL build testing Not applicable
Help/Support
Help/support interface required Make sure link to support/help page exist and is easy reachable.
Support documents planned(written) Make sure support documents are written and are correct.
Install/Upgrade
Feature upgrades/downgrades data as expected Not applicable
Does sync work across upgrades Not applicable
Requires install testing Default installation with Test Build
Affects first-run or onboarding To be tested
Enterprise
Enterprise administration Not applicable
Network proxies/autoconfig Not applicable
ESR behavior changes Not applicable
Locked preferences Not applicable
Data Monitoring
Temporary or permanent telemetry monitoring List of error conditions to monitor
Telemetry correctness testing To be tested
Server integration testing To be tested
Offline and server failure testing To be tested
Load testing Not applicable
Add-ons
Addon API required? Not applicable
Comprehensive API testing Not applicable
Permissions Not applicable
Testing with existing/popular addons To be tested
Security
3rd-party security review Not applicable
Privilege escalation testing Not applicable
Fuzzing Not applicable
Web Compatibility
Testing against target sites To be tested
Survey of many sites for compatibility To be tested
Interoperability
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS Desktop only
Interaction of this feature with other browser features Possible testing with other browsers

Test suite

Full Test suite - [TestRail]

Bug Work

Tracking bug – meta bug 1277346 [3]
Tracking bug - bug 1282484 [4]

Bug fix verification
Bug No Summary Status Firefox Verion
[5] Empty Build
[6] Empty Build
Logged bugs

Sign off

Criteria

Check list

  • All test cases should be executed
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/QA)

Results

  • Link for the tests run
    • Daily Smoke, use template from link
    • Full Test suite, use template from link
    • Regression Test suite, if needed/available

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation
Test Cases Creation
Full Functional Tests Execution
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
QA Signoff - Nightly Release Email to be sent
QA Aurora - Full Testing
QA Signoff - Aurora Release Email to be sent
QA Beta - Full Testing
QA Signoff - Beta Release Email to be sent
Retrieved from "https://wiki.allizom.org/index.php?title=QA/Flash_SHIELD_Experiment&oldid=1163485"