QA/Theming/Testplan

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date	Version	Author	Description
02/07/2017	1.0	Krupa Raj	Created first draft
02/09/2017	1.1	Valentina V.	Updated

Overview

There is an effort to extend theming options present in Firefox today with a single, canonical and more capable alternative, based on WebExtensions. This document provides details about the testing efforts around this project.

Ownership

Firefox Dev Lead: Michael de Boer; irc nick :mikedeboer
Webextensions Dev: Matthew Wein; irc nick :mattw
QA Manager: Krupa Raj; irc nick :krupa
QA Lead: Victor Carciu; irc nick :victorc
Webextensions QA: Vasilica Mihasca; irc nick :vasilica_mihasca
Add-ons QA: Valentina Virlics; irc nick :ValentinaV

Entry Criteria

• QA has access to all relevant documents/PRDs/tracker bugs
• The testcases have been reviewed by devs
• Entry Criteria for Dev has been satisfied
• Changes have landed in the nightly build and/or AMO dev

Current Status

• The feature is under development

Exit Criteria

• All the bugs have been triaged
• All P1 and P2 bugs have been resolved fixed
• All resolved bugs are verified
• The find/fixed rate is going down
• QA has had enough time to test the feature

Testing summary

Scope of Testing

In Scope

QA will test the following features in detail:

• Submission of static themes and themextensions
• Validation of static themes and themextensions
• Listing of LWTs and themextensions indistinctly (on AMO)
• Management of static themes and themextensions
• Install and updates of static themes and themextensions
• Management of static themes and themextensions from the add-ons manager
• Using Firefox while a static theme/ themextension is installed
• Existing LWTs are migrated to themextension
• Support for old-style LWTs (old LWTs will still be served to older versions of Firefox)
• Retiring Complete themes (Firefox displays a useful error message)
• Chrome parity - a chrome theme will work seamlessly on Firefox

Out of Scope

• Testing of individual themextensions and their functionality
• Search for themes and themextensions in Addons Manager
• Load testing
• Performance testing

Requirements for testing

Environments

OSes covered: Windows, MacOSX, Linux 
Devices covered: Desktop, Android

Channel dependent settings (configs) and environment setups

Nightly

text

Aurora

text

Beta

text

Post Beta / Release

text

Test Strategy

Test Objectives

Ref	Function	Test Objective	Evaluation Criteria	Test Type	Owners
1	AMO - Public pages	Test static themes/themextenions listed in AMO public pages	Users can browse, search and install static themes/themextenions on AMO	Manual and Automation	QA Team
2	AMO - Devhub pages	Submission & Management of Static themes	Users can successfully submit, edit, update & delete their static themes	Manual and Automation	QA Team
3	AMO - Linter	Validation of static themes and themextensions	Linter is updated to successfully validate the new static themes and themextensions	Manual	QA Team
4	AMO - Reviewer pages	Review of Static themes/themextensions	Reviewers can view, approve and reject static themes	Manual	QA Team
5	AMO - Admin	Admin access to static themes and themextensions	All the admin functionality is still applicable to static themes/ themextensions	Manual	QA Team
6	Add-ons Manager	All existing Add-ons manager functionality is retained	Users can browse, search and manage their installed static themes/themextensions via the Add-ons manager	Manual	QA Team
7	Backward Compatibility	Support for existing LWTs	Users can successfully install LWTs on all versions of Firefox	Manual	QA Team
8	Firefox with static theme/themextensions installed	Regression testing for Firefox with static theme/themextensions installed	There is no degradation of Firefox behavior with static theme/themextensions installed	Manual	QA Team

Builds

This section should contain links for builds with the feature -

• Links for Nightly builds
• Links for Aurora builds
• Links for Beta builds

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase	Start Date	End Date
Start project
Study documentation/specs received from developers
QA - Test plan creation	02/07/2017
QA - Test cases/Env preparation
QA - Nightly Testing
QA - Aurora Testing
QA - Beta Testing
Release Date	Firefox 55

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process	Tool
Test plan creation	Mozilla wiki
Test case creation	TestRail/ Google docs
Test case execution	TestRail
Bugs management	Bugzilla / Github

Status

Overview

Track the dates and build number where feature was released to Nightly
Track the dates and build number where feature was merged to Aurora
Track the dates and build number where feature was merged to Release/Beta

Risk analysis

Identify the high-risk assumptions
Identify existing bugs on the feature with high risk
Identify if other areas are affected by the fix

References

• PRD : https://docs.google.com/document/d/1ebB02HHc5Ni_ZyUwA41vJ9OlZsTEAMFOpXCaCqvuGTE/edit?ts=582dc369#heading=h.etihl2gdfzx6
• Engineering planning doc: https://docs.google.com/document/d/1ueD6V7aLLTuc1GAOxxQYcwl2HR-k62HHu3q8knTJ4FU/edit#
• AMO PRD: https://docs.google.com/document/d/1d9acBumpdK87sYg7Ah98BEKvL7b4tuO-RDpgqbm7FTs/edit#heading=h.ph0yhtsx9uv6
• Meta bug: Bug 1330328

Testcases

Test cases are being figured out at https://public.etherpad-mozilla.org/p/themeing

Overview

Summary of testing scenarios

Test Areas

We need to decide if this section is useful

Test Areas	Covered	Details
Private Window
Multi-Process Enabled
Multi-process Disabled
Theme (high contrast)
UI
Mouse-only operation
Keyboard-only operation
Display (HiDPI)
Interraction (scroll, zoom)
Usable with a screen reader		e.g. with NVDA
Usability and/or discoverability testing		Is this feature user friendly
RTL build testing
Help/Support
Help/support interface required		Make sure link to support/help page exist and is easy reachable.
Support documents planned(written)		Make sure support documents are written and are correct.
Install/Upgrade
Feature upgrades/downgrades data as expected
Does sync work across upgrades
Requires install testing		separate feature/application installation needed (not only Firefox)
Affects first-run or onboarding		Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
Does this affect partner builds? Partner build testing		yes/no options, add comment with details about who will lead testing
Enterprise		Raise up the topic to developers to see if they are expecting to work different on ESR builds
Enterprise administration
Network proxies/autoconfig
ESR behavior changes
Locked preferences
Data Monitoring
Temporary or permanent telemetry monitoring		List of error conditions to monitor
Telemetry correctness testing
Server integration testing
Offline and server failure testing
Load testing
Add-ons		If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required?
Comprehensive API testing
Permissions
Testing with existing/popular addons
Security		Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
3rd-party security review
Privilege escalation testing
Fuzzing
Web Compatibility		depends on the feature
Testing against target sites
Survey of many sites for compatibility
Interoperability		depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS
Interaction of this feature with other browser features

Test suite

Full Test suite - Link with the gdoc, follow the format from link
Smoke Test suite - Link with the gdoc, follow the format from link
Regression Test suite - Link with the gdoc - if available/needed.

Bug Work

Tracker – Bug 1330328

Bug fix verification

Bug No	Summary	Status	Firefox Verion
123	bug summary	NEW	Nighly 45

Logged bugs

Bug 111111
Bug 211111

Sign off

Results

Nightly testing

List of OSes that will be covered by testing

• Link for the tests run
  • Daily Smoke, use template from link
  • Full Test suite, use template from link
  • Regression Test suite, if needed/available

Merge to Aurora Sign-off List of OSes that will be covered by testing

• Link for the tests run
  • Full Test suite

Checklist

Exit Criteria	Status	Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation
Test Cases Creation
Full Functional Tests Execution
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
QA Signoff - Nightly Release		Email to be sent
QA Aurora - Full Testing
QA Signoff - Aurora Release		Email to be sent
QA Beta - Full Testing
QA Signoff - Beta Release		Email to be sent