QA/WebVR: Difference between revisions

From MozillaWiki
< QA
Jump to navigation Jump to search
(Polishing up the document.)
 
(47 intermediate revisions by 2 users not shown)
Line 1: Line 1:
'''Approvals Required / Received'''
The following individuals are required to/have approved this Test Plan:
{| class="wikitable"
|-
! Name !! Title !! Department !! Approval Date !! Method
|-
|  || QA Manager || Product Integrity || Date || Email
|-
|  || Software Engineer || Engineering || Date || Email
|-
|  || EPM || Product Management || Date || Email
|}
'''Revision History'''
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded).  The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
{| class="wikitable" style="width:60%"
|-
! Date !! Version !! Author !! Description
|-
| 07/12/2017 || 1.0 || Cristian Comorasu || Created first draft
|}
= Overview =
= Overview =
== Purpose ==
== Purpose ==
* The purpose of this wiki is to serve as a general test plan for the WebVR API.
The purpose of this document is to describe the WebVR feature from a testing perspective, including the following details:
* See https://wiki.mozilla.org/QA/MozVR for other test plans.
 
*Scope, focus areas and objectives of testing
*Owners and points of contact for areas of responsibility
*Strategy and types of testing
*The entry and exit criteria
*The basis of the test estimates
*Any risks, issues, assumptions and test dependencies
*The test schedule and major milestones
*The test deliverables
 
See https://wiki.mozilla.org/QA/MozVR for other test plans.
 
== Scope ==
This wiki details the testing that will be performed by the project team for the webVR project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
* What will be tested
* How testing will be performed


== Ownership ==
== Ownership ==
Developer contacts: [mailto:kgilbert@mozilla.com Kearwood Kip Gilbert] <br />
Developer contacts: [mailto:kgilbert@mozilla.com Kearwood Kip Gilbert] <br />
QA: [mailto:nhirata@mozilla.com Naoki Hirata (:nhirata_)], [mailto:cornel.ionce@softvision.ro Cornel Ionce (:cornel_ionce)] <br />
QA contacts: [mailto:cristian.comorasu@softvision.ro Cristian Comorasu (:CristiComo)], [mailto:cornel.ionce@softvision.ro Cornel Ionce (:cornel_ionce)] <br />
 
= Testing Summary =  
= Testing summary =  
== Scope of testing ==
== Scope of Testing ==
=== In Scope ===
=== In Scope ===
The current testing scope is to ensure that the WebVR API:
The current testing scope is to ensure that the WebVR:
# does not have regressions in performance or latency
# does not have regressions in performance or latency
# is stable
# is stable
# is reliable ( visibility )
# is reliable ( visibility )
=== Out of Scope ===
Detail what is out of scope from a testing perspective for the project team. Note: if usability testing is not in the scope of testing feature.


= Requirements for testing =
= Requirements for testing =
Line 20: Line 65:
Testing will be performed on following OSes (x64 infrastructures):
Testing will be performed on following OSes (x64 infrastructures):
* Windows 10
* Windows 10
* Windows 8.1


== Hardware ==
== Hardware ==
Line 25: Line 71:
* Graphics
* Graphics
** GTX 970 equivalent or greater
** GTX 970 equivalent or greater
** AMD 290 equivalent or greater
** AMD 290 R9 equivalent or greater
** Quadro k600 or newer
* CPU
* CPU
** Intel i5-4590 equivalent or greater
** Intel i5-4590 equivalent or greater
Line 36: Line 81:


== Devices ==
== Devices ==
* Oculus Development Kit 2 with positional tracker
*Oculus Rift CV1 (for pc) + Oculus Touch Controlllers
** Note: soon to change to CV1
*HTC Vive (for pc) + Vive Controllers
Testing will be performed on a machine with the following specifications:
** GPU: MSI GeForce® GTX 1050 2GT OC, 2GB GDDR5
** CPU: AMD FX-8320 3.50 GHz
** RAM: 16 gb DDR3 @ 1600mhz
 
== Channel dependent settings (configs) and environment setups ==
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
 
=== Nightly ===
<div class="mw-collapsible-content">
text
</div>
 
=== Beta ===
<div class="mw-collapsible-content">
text
</div>
 
</div>
 
= Test Strategy =
== Risk Assessment and Coverage ==
 
{| class="wikitable"
|-
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score
|-
| RAC-1 || Risk description 1 || TO-1 || 2-Moderate || 1-Unlikely || 3-High || 6
|-
| RAC-2 || Risk description 2 || TO-1 || 3-High || 3-Almost Certain || 3-High || 27
|-
| RAC-3 || Risk description 3 || TO-2 || 2-Moderate || 2-Possible || 3-High || 12
|}
 
'''Values:'''
 
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High'''
 
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''
 
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''
 
'''Impact Score Breakdown:'''
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.


= Test Strategy =
== Test Objectives ==
== Test Objectives ==
Following table contains main features to be tested:
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.
This could be documented in bullet form or in a table similar to the one below.
 
{| class="wikitable"
{| class="wikitable"
|-
|-
! Ref !! Function !! Test Objective !! Evaluation Criteria
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners
|-
|-
| 1 || Basic scenarios || TBD || TBD
| 1 || WebVR || The visual efects and user happy flow. || The feature is smooth and visualy reliable. || Manual,Regression, Performance, Usability || RAC-1, RAC-2, RAC-3 || Desktop Team
|-
|-
| 2 || || || || || ||
|}
|}


== Test Types ==  
== Builds ==
{|class="wikitable" style="width:60%"
* [https://nightly.mozilla.org/ Nightly Firefox]
** Archives of older builds are listed here : http://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/
* [https://www.mozilla.org/en-US/firefox/55.0beta/releasenotes/ Beta Firefox]
** Archives of older builds are listed here : https://archive.mozilla.org/pub/firefox/candidates/
 
== Test Execution Schedule ==
The following table identifies the anticipated testing period available for test execution.
{| class="wikitable" style="width:60%"
|-
|-
! Type of Testing !! Required (Yes/No) !! Owners !! Details/Comments
! Project phase !! Start Date !! End Date
|-
|-
| Manual
| Start project
|style="text-align:center;" | Yes
|style="text-align:center;" | 07.12.2017 || TBD
|style="text-align:center;" | Moz QA Rel team ||
|-
|-
| Automation
| Study documentation/specs received from developers
|style="text-align:center;" |
|style="text-align:center;" | 07.12.2017 || 07.13.2017
|style="text-align:center;" | ||
|-
| QA - Test plan creation
|style="text-align:center;" | 07.12.2017 || 07.12.2017
|-
|-
| Performance
| QA - Test cases/Env preparation
|style="text-align:center;" | No ||  ||
|style="text-align:center;" | 07.13.2017 || 07.14.2017
|-
|-
| Security
| QA - Nightly Testing
|style="text-align:center;" | No ||  ||
|style="text-align:center;" | Missed || Missed
|-
|-
| Usability
| QA - Beta Testing
|style="text-align:center;" | No ||  ||
|style="text-align:center;" | 07.14.2017 || 07.21.2017
|-
|-
| Metrics/Telemetry
| Release Date
|style="text-align:center;" | TBD|| ||
|style="text-align:center;" |TBD || TBD
|}
|}
== Builds ==
This section should contain links for builds with the feature -
Nightly builds with can be found at [http://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/ link]


== Testing Tools ==
== Testing Tools ==
Detail the tools to be used for testing, for example see the following table:
Detail the tools to be used for testing, for example see the following table:
{| class="wikitable" style="width:60%"
{| class="wikitable" style="width:50%"
|-
|-
! Process !! Tool
! Process !! Tool
Line 89: Line 185:
| Test plan creation || Mozilla wiki
| Test plan creation || Mozilla wiki
|-
|-
| Test case creation || Google docs
| Test case creation || [https://testrail.stage.mozaws.net/index.php TestRail]/ Google docs
|-
|-
| Test case execution || Google docs
| Test case execution || [https://testrail.stage.mozaws.net/index.php TestRail]
|-
|-
| Bugs management || Bugzilla
| Bugs management || Bugzilla
|}
|}


= Status =
= Status =  
== Overview ==
== Overview ==
<small>'''<font color="green">[LANDED]</font>'''</small> Nightly 46: first landed on - '''2016.01.05''
Track the dates and build number where feature was released to Nightly (TBD)
Track the dates and build number where feature was merged to Release/Beta (TBD)


== Risk analysis ==
- No Risks found at the moment


= References =
= References =
Meta: [https://bugzilla.mozilla.org/show_bug.cgi?id=1218482 Bug 1218482 - Enable WebVR By Default on all platforms]
* List and links for specs
  List and links for available specs - documents, user stories, specifications (TBD)
* Meta bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1250244
= Testcases =
== Test Areas ==
{| class="wikitable" style="width:80%"
|-
! Test Areas !! Covered !! Details
|-
| Private Window
|style="text-align:center;" | YES || -
|-
| Multi-Process Enabled
|style="text-align:center;" | YES || -
|-
| Multi-process Disabled
|style="text-align:center;" | YES || -
|-
| Theme (high contrast)
|style="text-align:center;" | NO || -
|-
| '''UI'''
||  ||
|-
| Mouse-only operation 
|style="text-align:center;" | NO || -
|-
| Keyboard-only operation 
|style="text-align:center;" | NO || -
|-
| Display (HiDPI)
|style="text-align:center;" | YES || TBD
|-
| Interaction (scroll, zoom)
|style="text-align:center;" | NO || -
|-
| Usable with a screen reader 
|style="text-align:center;" | NO || -
|-
| Usability and/or discoverability testing 
|style="text-align:center;" | YES || TBD
|-
| RTL build testing 
|style="text-align:center;" | NO || -
|-
| '''Help/Support'''
||  ||
|-
| Help/support interface required 
|style="text-align:center;" | YES || TBD
|-
| Support documents planned(written) 
|style="text-align:center;" | YES || -


= Test cases =
|-
== Test suites ==
| '''Install/Upgrade'''
Full Test Suite - can be reached at [https://goo.gl/VqKbTV this link]
||  ||
|-
| Feature upgrades/downgrades data as expected 
|style="text-align:center;" | YES || TBD
|-
| Does sync work across upgrades 
|style="text-align:center;" | NO || TBD
|-
| Requires install testing   
|style="text-align:center;" | YES || TBD
|-
| Affects first-run or onboarding   
|style="text-align:center;" | NO || TBD
|-
| Does this affect partner builds? Partner build testing 
|style="text-align:center;" | NO || -


= Bugs Work =
|-
Tracking bug: Meta bug [https://bugzilla.mozilla.org/show_bug.cgi?id=1218482 1218482]
| ''' Enterprise '''
||  ||  Raise up the topic to developers to see if they are expecting to work different on ESR builds
|-
| Enterprise administration 
|style="text-align:center;" | NO || -
|-
| Network proxies/autoconfig 
|style="text-align:center;" | NO || -
|-
| ESR behavior changes 
|style="text-align:center;" | NO || -
|-
| Locked preferences 
|style="text-align:center;" | NO || -


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
|-
| ''' Data Monitoring '''
||  ||
|-
| Temporary or permanent telemetry monitoring 
|style="text-align:center;" | NO || -
|-
| Telemetry correctness testing 
|style="text-align:center;" | NO || -
|-
| Server integration testing 
|style="text-align:center;" | NO || -
|-
| Offline and server failure testing 
|style="text-align:center;" | NO || -
|-
| Load testing 
|style="text-align:center;" | NO || -
 
|-
| ''' Add-ons '''
||  ||
|-
| Addon API required? 
|style="text-align:center;" | NO || -
|-
| Comprehensive API testing 
|style="text-align:center;" | NO || -
|-
| Permissions 
|style="text-align:center;" | YES || TBD
|-
| Testing with existing/popular addons
|style="text-align:center;" | NO || -
 
|-
| ''' Security '''
||  || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
|-
| 3rd-party security review 
|style="text-align:center;" | YES || TBD
|-
| Privilege escalation testing
|style="text-align:center;" | YES || TBD
|-
| Fuzzing 
|style="text-align:center;" | YES || TBD
 
|-
| ''' Web Compatibility '''
||  || depends on the feature
|-
| Testing against target sites 
|style="text-align:center;" | YES || TBD
|-
| Survey of many sites for compatibility 
|style="text-align:center;" | YES || TBD


====== Bug fix verification ======
<div class="mw-collapsible-content">
{| class="wikitable"
|-
|-
! Bug No !! Summary !! Status !! Firefox Version
| ''' Interoperability  '''
||  || depends on the feature
|-
|-
| [https://bugzilla.mozilla.org/show_bug.cgi?id=1218482 1218482] ||[webvr] Enable WebVR By Default on all platforms ||Verified - Fixed || Firefox 46
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.  
|style="text-align:center;" | YES || TBD
|-
|-
| [https://bugzilla.mozilla.org/show_bug.cgi?id=1182048 1182048] ||Make webvr and e10s work together ||Verified - Fixed|| Firefox 46
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS 
|style="text-align:center;" | NO || -
|-
|-
| Interaction of this feature with other browser features 
|style="text-align:center;" | NO || -
|}
|}
== Test suite ==
Full Test suite - https://goo.gl/W3bmYm
Smoke Test suite - https://goo.gl/W3bmYm
Regression Test suite - https://goo.gl/W3bmYm
= Bug Work =
Meta bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1250244
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1250244 1250244] )======
<div class="mw-collapsible-content">
<bugzilla>
    {
        "blocks":[1250244],
        "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
    }
</bugzilla>
</div>
</div>
</div>


====== Logged bugs ======
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
====== Bug fix verification ======
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
<bugzilla>
    {
        "blocks":[12345],
        "resolution":"FIXED",
        "include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"
    }
</bugzilla>
</div>
</div>
</div>
</div>


= Sign off =
= Sign off =
== Criteria ==
== Criteria ==
Checklist
* All test cases should be executed
* All test cases should be executed
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)


== Results ==
== Results ==
'''Merge to Aurora Sign-off'''
'''Nightly testing(missed)'''<br />
* {{done|}} mid-Aurora 46 ''(2016-02-15)''.
<br />
** There were no blockers found while testing this feature.
 
** Manual testing covered Windows 10 (x64) and Mac OS X 10.10.5.
'''Merge to Beta Sign-off'''<br />
'''Merge to Beta Sign-off'''  
List of OSes that will be covered by testing:<br />
* {{block|}} Pre-beta 46 ''(2016-03-04)''.
*Windows 10
** Crash 1239055 is severely affecting the feature (considered a blocker for the release).
*Windows 8.1
** Manual testing covered Windows 10 (x64) on two machines.
*Windows 7
*Link for the tests run - N/A
** Full Test suite - In progress
 
== Checklist ==
{| class="wikitable" style="width:60%"
|-
! Exit Criteria !! Status !! Notes/Details
|-
|  Testing Prerequisites (specs, use cases)  
| style="text-align:center;" | 
| style="text-align:center;" |
|-
|  Testing Infrastructure setup
|style="text-align:center;" | DONE||
|-
|  Test Plan Creation
| style="text-align:center;" | DONE ||
|-
|  Test Cases Creation
|style="text-align:center;" | DONE ||
|-
|  Automation Coverage ||
|style="text-align:center;" |
|-
|  Performance Testing
|style="text-align:center;" | DONE ||
|-
|  All Defects Logged || ||
|-
|  Critical/Blockers Fixed and Verified || ||
|-
|  Metrics/Telemetry||
|style="text-align:center;" |
|-
|  Basic/Core functionality Nightly testing
|style="text-align:center;" | 
|style="text-align:center;" | 
|-
|  QA mid-Nightly Signoff||
|style="text-align:center;" | Missed
|-
|  QA Nightly - Full Testing
|style="text-align:center;" | Missed ||
|-
|  QA pre-Beta Signoff||
|style="text-align:center;"| Missed
|-
|  QA Beta - Full Testing
|style="text-align:center;" |  ||
|-
|  QA pre-Release Signoff  || DONE
|style="text-align:center;" | Email  sent
|}

Latest revision as of 11:39, 24 July 2017

Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name Title Department Approval Date Method
QA Manager Product Integrity Date Email
Software Engineer Engineering Date Email
EPM Product Management Date Email


Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
07/12/2017 1.0 Cristian Comorasu Created first draft

Overview

Purpose

The purpose of this document is to describe the WebVR feature from a testing perspective, including the following details:
  • Scope, focus areas and objectives of testing
  • Owners and points of contact for areas of responsibility
  • Strategy and types of testing
  • The entry and exit criteria
  • The basis of the test estimates
  • Any risks, issues, assumptions and test dependencies
  • The test schedule and major milestones
  • The test deliverables
See https://wiki.mozilla.org/QA/MozVR for other test plans.

Scope

This wiki details the testing that will be performed by the project team for the webVR project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • What will be tested
  • How testing will be performed

Ownership

Developer contacts: Kearwood Kip Gilbert
QA contacts: Cristian Comorasu (:CristiComo), Cornel Ionce (:cornel_ionce)

Testing summary

Scope of Testing

In Scope

The current testing scope is to ensure that the WebVR:

  1. does not have regressions in performance or latency
  2. is stable
  3. is reliable ( visibility )

Out of Scope

Detail what is out of scope from a testing perspective for the project team. Note: if usability testing is not in the scope of testing feature.

Requirements for testing

Environments

Testing will be performed on following OSes (x64 infrastructures):

  • Windows 10
  • Windows 8.1

Hardware

The minimum system requirements for Windows OS are:

  • Graphics
    • GTX 970 equivalent or greater
    • AMD 290 R9 equivalent or greater
  • CPU
    • Intel i5-4590 equivalent or greater
    • AMD FX 8320 or greater
  • 8GB+ RAM
  • Compatible HDMI 1.3 video output
  • 2x USB 3.0 ports
  • Windows 7 SP1 or newer, plus the DirectX platform update

Devices

  • Oculus Rift CV1 (for pc) + Oculus Touch Controlllers
  • HTC Vive (for pc) + Vive Controllers

Testing will be performed on a machine with the following specifications:

    • GPU: MSI GeForce® GTX 1050 2GT OC, 2GB GDDR5
    • CPU: AMD FX-8320 3.50 GHz
    • RAM: 16 gb DDR3 @ 1600mhz

Channel dependent settings (configs) and environment setups

Nightly

text

Beta

text

Test Strategy

Risk Assessment and Coverage

ID Description / Threat Description Covered by Test Objective Magnitude Probability Priority Impact Score
RAC-1 Risk description 1 TO-1 2-Moderate 1-Unlikely 3-High 6
RAC-2 Risk description 2 TO-1 3-High 3-Almost Certain 3-High 27
RAC-3 Risk description 3 TO-2 2-Moderate 2-Possible 3-High 12

Values:

  • Magnitude: 1- Low , 2-Moderate, 3-High
  • Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
  • Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

  • An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
  • An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
  • An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref Function Test Objective Evaluation Criteria Test Type RAC Owners
1 WebVR The visual efects and user happy flow. The feature is smooth and visualy reliable. Manual,Regression, Performance, Usability RAC-1, RAC-2, RAC-3 Desktop Team
2

Builds

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 07.12.2017 TBD
Study documentation/specs received from developers 07.12.2017 07.13.2017
QA - Test plan creation 07.12.2017 07.12.2017
QA - Test cases/Env preparation 07.13.2017 07.14.2017
QA - Nightly Testing Missed Missed
QA - Beta Testing 07.14.2017 07.21.2017
Release Date TBD TBD

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail/ Google docs
Test case execution TestRail
Bugs management Bugzilla

Status

Overview

Track the dates and build number where feature was released to Nightly (TBD)
Track the dates and build number where feature was merged to Release/Beta (TBD)


References

  • List and links for specs
 List and links for available specs - documents, user stories, specifications (TBD)

Testcases

Test Areas

Test Areas Covered Details
Private Window YES -
Multi-Process Enabled YES -
Multi-process Disabled YES -
Theme (high contrast) NO -
UI
Mouse-only operation NO -
Keyboard-only operation NO -
Display (HiDPI) YES TBD
Interaction (scroll, zoom) NO -
Usable with a screen reader NO -
Usability and/or discoverability testing YES TBD
RTL build testing NO -
Help/Support
Help/support interface required YES TBD
Support documents planned(written) YES -
Install/Upgrade
Feature upgrades/downgrades data as expected YES TBD
Does sync work across upgrades NO TBD
Requires install testing YES TBD
Affects first-run or onboarding NO TBD
Does this affect partner builds? Partner build testing NO -
Enterprise Raise up the topic to developers to see if they are expecting to work different on ESR builds
Enterprise administration NO -
Network proxies/autoconfig NO -
ESR behavior changes NO -
Locked preferences NO -
Data Monitoring
Temporary or permanent telemetry monitoring NO -
Telemetry correctness testing NO -
Server integration testing NO -
Offline and server failure testing NO -
Load testing NO -
Add-ons
Addon API required? NO -
Comprehensive API testing NO -
Permissions YES TBD
Testing with existing/popular addons NO -
Security Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
3rd-party security review YES TBD
Privilege escalation testing YES TBD
Fuzzing YES TBD
Web Compatibility depends on the feature
Testing against target sites YES TBD
Survey of many sites for compatibility YES TBD
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. YES TBD
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS NO -
Interaction of this feature with other browser features NO -

Test suite

Full Test suite - https://goo.gl/W3bmYm 
Smoke Test suite - https://goo.gl/W3bmYm
Regression Test suite - https://goo.gl/W3bmYm

Bug Work

Meta bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1250244

Logged bugs ( blocking 1250244 )
Full Query
ID Priority Component Assigned to Summary Status Target milestone
1198518 -- Graphics gfrolov [webvr] add support for OSVR RESOLVED mozilla49
1256444 -- WebVR [webvr] Enable WebVR API in Release RESOLVED ---
1276066 -- Graphics :kip (Kearwood Gilbert) Add Union with "components" member to math classes to enable array access to members RESOLVED mozilla49
1276359 -- Graphics: Layers :kip (Kearwood Gilbert) Compiling D3D11 Shaders Fails RESOLVED mozilla49
1276811 -- Graphics: Layers :kip (Kearwood Gilbert) [webvr] Bug 1276811 - Enable TextureClient to be used without CompositableForwarder RESOLVED mozilla50
1284292 -- Graphics :kip (Kearwood Gilbert) [webvr] Allow GLScreenBuffer::CreateFactory to work without a CompositableForwarder RESOLVED mozilla50
1284324 -- Graphics :kip (Kearwood Gilbert) [webvr] Implement Canvas Layer Mirrors RESOLVED mozilla50
1284357 -- Graphics :kip (Kearwood Gilbert) [webvr] Implement Navigator.activeVRDisplays RESOLVED mozilla51
1284984 -- WebVR :kip (Kearwood Gilbert) [webvr] Add Gamepad.displayID RESOLVED ---
1295951 -- Graphics :kip (Kearwood Gilbert) Crash in SnowWhiteKiller::Trace RESOLVED mozilla51
1296381 -- Graphics: Layers Benoit Girard (:BenWa) FEATURE_FAILURE_D3D11_VR_BUFFER was not removed RESOLVED mozilla51
1325810 P2 Graphics :kip (Kearwood Gilbert) Graphics is lagged when moving mouse in circular motion RESOLVED mozilla53
1383094 -- WebVR Unplugging your VR device displays no error prompt RESOLVED ---
1383095 P3 WebVR Misleading prompt RESOLVED ---
1383097 P1 WebVR Freeze in https://aframe.io/a-painter/ RESOLVED ---
1383099 -- WebVR Closing the VR webpage causes the VR to remain frozen VERIFIED ---
1383104 P2 WebVR Refreshing the VR page causes the VR device to have jerky movements or crash RESOLVED ---
1383105 -- WebVR Game freezes in the VR device VERIFIED ---
1383106 P1 WebVR The device glitches on some demos RESOLVED ---
1383107 -- WebVR :kip (Kearwood Gilbert) [VR] Browser crash if the user opens a different page in a new tab RESOLVED ---
1383110 -- WebVR Daosheng Mu[:daoshengmu] Controllers are positioned inside the headset VERIFIED mozilla56

21 Total; 0 Open (0%); 18 Resolved (85.71%); 3 Verified (14.29%);


Bug fix verification
Full Query
ID Priority Component Assigned to Summary Status Resolution Target milestone
15069 P3 Networking jefft File type url not correctly parsed and created VERIFIED FIXED M13

1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);

Sign off

Criteria

Checklist

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing(missed)

Merge to Beta Sign-off
List of OSes that will be covered by testing:

  • Windows 10
  • Windows 8.1
  • Windows 7
  • Link for the tests run - N/A
    • Full Test suite - In progress

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup DONE
Test Plan Creation DONE
Test Cases Creation DONE
Automation Coverage
Performance Testing DONE
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
Basic/Core functionality Nightly testing
QA mid-Nightly Signoff Missed
QA Nightly - Full Testing Missed
QA pre-Beta Signoff Missed
QA Beta - Full Testing
QA pre-Release Signoff DONE Email sent
Retrieved from "https://wiki.allizom.org/index.php?title=QA/WebVR&oldid=1176369"