QA/WebVR: Difference between revisions
| Line 9: | Line 9: | ||
= Testing Summary = | = Testing Summary = | ||
== Introduction == | |||
This is a hybrid test plan of [https://code.google.com/archive/p/test-analytics/wikis/AccExplained.wiki ACC] and traditional test plan in regards to the [https://mozvr.github.io/webvr-spec/ WebVR API] | |||
== Scope of testing == | == Scope of testing == | ||
=== In Scope === | === In Scope === | ||
Revision as of 19:29, 22 April 2016
Overview
Purpose
- The purpose of this wiki is to serve as a general test plan for the WebVR API.
- See https://wiki.mozilla.org/QA/MozVR for other test plans.
Ownership
Developer contacts: Kearwood Kip Gilbert
QA: Naoki Hirata (:nhirata_), Cornel Ionce (:cornel_ionce)
Testing Summary
Introduction
This is a hybrid test plan of ACC and traditional test plan in regards to the WebVR API
Scope of testing
In Scope
The current testing scope is to ensure that the WebVR API:
- does not have regressions in performance or latency
- is stable
- is reliable ( visibility )
Requirements for testing
Environments
Testing will be performed on following OSes (x64 infrastructures):
- Windows 10
Hardware
The minimum system requirements for Windows OS are:
- Graphics
- GTX 970 equivalent or greater
- AMD 290 equivalent or greater
- Quadro k600 or newer
- CPU
- Intel i5-4590 equivalent or greater
- AMD FX 8320 or greater
- 8GB+ RAM
- Compatible HDMI 1.3 video output
- 2x USB 3.0 ports
- Windows 7 SP1 or newer, plus the DirectX platform update
Setup
- https://docs.google.com/document/d/16z5cAl21yeuaH_kYJBAs--g8HgXHj89snJI_8Py4VXQ/edit#
- Note: needs updating
Devices
- Oculus Development Kit 2 with positional tracker
- Note: soon to change to CV1
Test Strategy
Test Objectives
Following table contains main features to be tested:
| Ref | Function | Test Objective | Evaluation Criteria |
|---|---|---|---|
| 1 | Basic scenarios | TBD | TBD |
Test Types
| Type of Testing | Required (Yes/No) | Owners | Details/Comments |
|---|---|---|---|
| Manual | Yes | Moz QA Rel team | |
| Automation | |||
| Performance | No | ||
| Security | No | ||
| Usability | No | ||
| Metrics/Telemetry | TBD |
Builds
This section should contain links for builds with the feature -
Nightly builds with can be found at link
Testing Tools
Detail the tools to be used for testing, for example see the following table:
| Process | Tool |
|---|---|
| Test plan creation | Mozilla wiki |
| Test case creation | Google docs |
| Test case execution | Google docs |
| Bugs management | Bugzilla |
Status
Overview
'[LANDED] Nightly 46: first landed on - 2016.01.05
Risk analysis
- No Risks found at the moment
References
Meta: Bug 1218482 - Enable WebVR By Default on all platforms
Test cases
Test suites
Full Test Suite - can be reached at this link
Bugs Work
Tracking bug: Meta bug 1218482
Bug fix verification
| Bug No | Summary | Status | Firefox Version |
|---|---|---|---|
| 1218482 | [webvr] Enable WebVR By Default on all platforms | Verified - Fixed | Firefox 46 |
| 1182048 | Make webvr and e10s work together | Verified - Fixed | Firefox 46 |
Logged bugs
Sign off
Criteria
- All test cases should be executed
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
Merge to Aurora Sign-off
- [DONE] mid-Aurora 46 (2016-02-15).
- There were no blockers found while testing this feature.
- Manual testing covered Windows 10 (x64) and Mac OS X 10.10.5.
Merge to Beta Sign-off
- [BLOCKED] Pre-beta 46 (2016-03-04).
- Crash 1239055 is severely affecting the feature (considered a blocker for the release).
- Manual testing covered Windows 10 (x64) on two machines.