QA/Theming/Testplan: Difference between revisions

From MozillaWiki
< QA
Jump to navigation Jump to search
No edit summary
Line 92: Line 92:
=== Nightly ===
=== Nightly ===
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
text
Set extensions.webextensions.themes.enabled to true
</div>
</div>


=== Aurora ===
=== Aurora ===
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
text
Set extensions.webextensions.themes.enabled to true
</div>
</div>


=== Beta ===
=== Beta ===
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
text
Set extensions.webextensions.themes.enabled to true
</div>
</div>


Revision as of 21:11, 6 March 2017

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
02/07/2017 1.0 Krupa Raj Created first draft
02/09/2017 1.1 Valentina V. Updated
02/09/2017 1.1 Carciu Victor Updated
03/03/2017 1.1 Carciu Victor Updated - Added test cases summaries under Testcases Overview , Added some new items under Scope of Testing

Overview

There is an effort to extend theming options present in Firefox today with a single, canonical and more capable alternative, based on WebExtensions. This document provides details about the testing efforts around this project.

Ownership

Firefox Dev Lead: Michael de Boer; irc nick :mikedeboer
Webextensions Dev: Matthew Wein; irc nick :mattw
QA Manager: Krupa Raj; irc nick :krupa
QA Lead: Victor Carciu; irc nick :victorc
Webextensions QA: Vasilica Mihasca; irc nick :vasilica_mihasca
Add-ons QA: Valentina Virlics; irc nick :ValentinaV

Entry Criteria

  • QA has access to all relevant documents/PRDs/tracker bugs
  • The testcases have been reviewed by devs
  • Feature has sufficient unit test coverage (determined by the devs)
  • Features landing in Firefox have regression test coverage. This can be accessed by
  • Entry Criteria for Dev has been satisfied
  • Changes have landed in the nightly build and/or AMO dev

Current Status

  • The feature is under development

Exit Criteria

  • All the bugs have been triaged
  • All P1 and P2 bugs have been resolved fixed
  • All resolved bugs are verified
  • The find/fixed rate is going down
  • QA has had enough time to test the feature

Testing summary

Scope of Testing

In Scope

QA will test the following features in detail:

  • Submission of static themes and themextensions
  • Validation of static themes and themextensions
  • Listing of LWTs and static themes indistinctly (on AMO)
  • Management of static themes and themextensions
  • Install and updates of static themes and themextensions
  • Management of static themes and themextensions from the add-ons manager
  • Using Firefox while a static theme/ themextension is installed
  • Support for old-style LWTs
  • Retiring Complete themes (Firefox displays a useful error message)
  • Chrome parity - a chrome theme will work seamlessly on Firefox
  • Android will support only actual LWTs
  • RTL testing for Theming

Out of Scope

  • Testing of individual themextensions and their functionality
  • Load testing
  • Performance testing

Requirements for testing

Environments

OSes covered: Windows, MacOSX, Linux 
Devices covered: Desktop, Android

Compatibility Matrix

Add-on Type Supported Firefox versions Supported platforms Other Browsers
LWTs (old style) All Desktop and Android Install not supported/appropriate messaging
Static Themes FF55 and greater Desktop Install not supported/appropriate messaging
Themextensions FF55 and greater Desktop Install not supported/appropriate messaging
Complete Themes supported until FF57 Desktop Install not supported/appropriate messaging

Note: For static themes on Android, there is appropriate error messaging if any user tries to install.

Channel dependent settings (configs) and environment setups

Nightly

Set extensions.webextensions.themes.enabled to true

Aurora

Set extensions.webextensions.themes.enabled to true

Beta

Set extensions.webextensions.themes.enabled to true

Post Beta / Release

text

Test Strategy

Test Objectives

Ref Function Test Objective Evaluation Criteria Test Type Owners
1 AMO - Public pages Test static themes/themextenions listed in AMO public pages Users can browse, search and install static themes/themextenions on AMO Manual and Automation QA Team
2 AMO - Devhub pages Submission & Management of Static themes Users can successfully submit, edit, update & delete their static themes Manual and Automation QA Team
3 AMO - Linter Validation of static themes and themextensions Linter is updated to successfully validate the new static themes and themextensions Manual QA Team
4 AMO - Reviewer pages Review of Static themes/themextensions Reviewers can view, approve and reject static themes Manual QA Team
5 AMO - Admin Admin access to static themes and themextensions All the admin functionality is still applicable to static themes/ themextensions Manual QA Team
6 Add-ons Manager All existing Add-ons manager functionality is retained Users can browse, search and manage their installed static themes/themextensions via the Add-ons manager Manual QA Team
7 Backward Compatibility Support for existing LWTs Users can successfully install LWTs on all versions of Firefox Manual QA Team
8 Firefox with static theme/themextensions installed Regression testing for Firefox with static theme/themextensions installed There is no degradation of Firefox behavior with static theme/themextensions installed Manual QA Team


Builds

This section should contain links for builds with the feature -

  • Links for Nightly builds
  • Links for Aurora builds
  • Links for Beta builds

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project
Study documentation/specs received from developers
QA - Test plan creation 02/07/2017
QA - Test cases/Env preparation
QA - Nightly Testing
QA - Aurora Testing
QA - Beta Testing
Release Date Firefox 55

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail/ Google docs
Test case execution TestRail
Bugs management Bugzilla / Github

Status

Overview

Track the dates and build number where feature was released to Nightly
Track the dates and build number where feature was merged to Aurora
Track the dates and build number where feature was merged to Release/Beta

Risk analysis

  • Android does not support the new static themes. A developer wanting to upload a new theme will have to submit two separate themes (old and new) if they want to be listed for desktop and mobile

References

Testcases

Test cases are being figured out at https://public.etherpad-mozilla.org/p/themeing

Overview

Summary of testing scenarios
  • Add-ons Manager
    • Check that LWTs can still be installed on older versions of Firefox
    • Check that LWTs can still be installed on the latest version of Firefox
    • Check that all installed/disabled themes are listed under Appearance tab
    • Check that installed LWTs continue to be listed after user upgrades to the latest version of Firefox
    • Check that the default theme cannot be deleted
    • Check that the user can switch back and forth between a static theme and a LWT
    • Check that the user can switch back and forth between a static theme and a default theme
    • Check that the user can disable an installed theme
    • Check that the user can remove an installed theme
    • Check that there is an undo option to cancel "Remove" action
    • Check that the theme is restored upon clicking the "Remove" action
    • Check that after a page refresh, all removed themes disappear from the Add-ons manager
    • Check that there is a 'More' link next to each of the installed theme
    • Check that for each listed theme we show - custom icon, excerpt of the theme description, and the theme name
    • Check that clicking the More link shows the following details - full theme description, Updated date and the name of the theme creator
    • Check that the Updated date listed in the add-ons manager is accurate
    • Check that user can switch between themes (and, the previously applied theme is disabled when a new one is applied and NOT removed)
    • Check that if the user installs 4 themes in succession, the latest one is the only one enabled (the rest are listed but disabled)
    • Check that by default only 30 themes can be installed
    • Check that each installed static theme has an associated custom icon which matches the theme image
    • Check that user can click on any of the listed themes in the add-ons manager
    • Check that the selected theme is highlighted with an orange bar
  • Firefox
    • Check that there is a warning shown to users if they try to install a theme from an unknown source
    • Check that there is no warning shown when users install a theme from a known source
    • Check that there is a notification by the browser when a theme is installed with the buttons - Undo and Manage Themes
    • Check that Firefox release comes with a default theme
    • Check that the Firefox default theme cannot be deleted
    • Check that only one theme can be applied at any time (we replace the old one with the new one)
    • Check the behavior if multiple themextensions try alter the browser chrome at the same time (same elements)
    • Check the behavior if multiple themextensions try alter the browser chrome at the same time (different elements)
    • Check that the applied static theme persists after a firefox upgrade
    • Check that the applied themextension persists after a firefox upgrade
    • Check that a disabled static theme remains disabled after a firefox upgrade
    • Check that a disabled themextension remains disabled after a firefox upgrade
    • Check that the applied static theme is specific to the current profile
    • Check that the installed themextension is specific to the current profile
    • Check behavior when window is resized with a static theme applied
    • Check behavior when window is resized with a themextension applied
    • Check switching to and from full-screen mode with a static theme installed
    • Check switching to and from full-screen mode with a themextension installed
      • Particularly concerned about user being kicked out of full-screen mode because some conditional in a themextension kicked in
    • Check that the profile always loads with the theme applied if there is any active theme
    • Check that firefox reloads with the static theme applied if there is a crash
    • Run all the regression tests for Graphics with a static theme ON
    • Run all the regression tests for Graphics with a themextension ON
    • Check that a user can apply a static theme with Side Tabs (add-on) installed
    • Check that a user can apply a themextension with Side Tabs (add-on) installed
    • Check that a user can apply a static theme with Tabs Grouping (add-on) installed
    • Check that a user can apply a themextension with Tabs Grouping (add-on) installed
    • If https://bugzilla.mozilla.org/show_bug.cgi?id=1319821 is implemented
      • check that static themes can apply multiple images(?)
    • Check if themes work as expected on RTL builds
    • Check if themextensions work as expected on RTL builds
  • Android:
    • Check that existing LWTs still work as expected
    • Check that there is a proper error message when an android user tries to install a static theme
  • AMO
    • Public pages
      • Check that LWTs and static themes are listed on the same page without any differentiation
      • Check that only old-style LWTs are listed for Android
      • Check that you can preview a static theme even when you have a theme installed
      • Check that you can preview a static theme even when you have dynamic webext installed
      • Check that search works for -
        • LWTs
        • static themes
        • themextensions
        • Listing pages
      • Check that user count for existing LWTs is not affected
      • Check that only old-style LWTs are listed for Android
      • What is the behavior when user is in /Android on desktop - what set of themese get listed?
    • Validator
      • Check that any property value that goes into a ‘url()’ CSS declaration is not reaching out of its sandbox of the extension package.
      • Check that a theme cannot include background scripts
      • Check that if the manifest.json includes a theme section, all other sections are ignored (we should throw a warning)
    • Devhub
      • Check that you can still submit old-style LWTs (for desktop and Android)
    • Reviewer tools
      • Check that a reviewer can view a theme under review
      • Check that a reviewer can approve a theme
      • Check that a reviewer can delete a theme
    • Admin tools
      • Check that a static theme can be blocked
      • Check that a themextension can be blocked

Test Areas

We need to decide if this section is useful

Test Areas Covered Details
Private Window Make sure that applying a theme or themextension works as expected in a private window
Multi-Process Enabled Themes and Themextension work as expected with e10s
Multi-process Disabled Themes and Themextension work as expected without e10s
UI
Mouse-only operation
Keyboard-only operation
Display (HiDPI)
Interraction (scroll, zoom) Scroll and zoom work as expected with a theme or themextension applied
Usability and/or discoverability testing Is this feature user friendly
RTL build testing Theme and themextensions work as expected on RTL builds
Help/Support
Help/support interface required Make sure link to support/help page exist and is easy reachable.
Support documents planned(written) Make sure support documents are written and are correct.
Install/Upgrade
Feature upgrades/downgrades data as expected
Does sync work across upgrades
Requires install testing separate feature/application installation needed (not only Firefox)
Affects first-run or onboarding Check if there is a marketing plan to highlight themes in the onboarding experience
Does this affect partner builds? Partner build testing yes/no options, add comment with details about who will lead testing
Enterprise Raise up the topic to developers to see if they are expecting to work different on ESR builds
Enterprise administration
Network proxies/autoconfig
ESR behavior changes
Locked preferences
Data Monitoring
Temporary or permanent telemetry monitoring List of error conditions to monitor
Telemetry correctness testing
Server integration testing
Offline and server failure testing
Load testing
Security Review should be done by the Security team
3rd-party security review
Privilege escalation testing
Fuzzing
Interoperability
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS Make sure that the old LWTs still work as expected on Android; Not applicable for iOS
Interaction of this feature with other browser features

Test suite

Full Test suite - Link with the gdoc, follow the format from link
Smoke Test suite - Link with the gdoc, follow the format from link
Regression Test suite - Link with the gdoc - if available/needed.

Bug Work

Tracker – Bug 1330328

Bug fix verification
Bug No Summary Status Firefox Verion
123 bug summary NEW Nighly 45
Logged bugs

Bug 111111
Bug 211111

Sign off

Results

Nightly testing

List of OSes that will be covered by testing

  • Link for the tests run
    • Daily Smoke, use template from link
    • Full Test suite, use template from link
    • Regression Test suite, if needed/available

Merge to Aurora Sign-off List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation
Test Cases Creation
Full Functional Tests Execution
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
QA Signoff - Nightly Release Email to be sent
QA Aurora - Full Testing
QA Signoff - Aurora Release Email to be sent
QA Beta - Full Testing
QA Signoff - Beta Release Email to be sent
Retrieved from "https://wiki.allizom.org/index.php?title=QA/Theming/Testplan&oldid=1164768"