Add-ons/QA/Testplan/about:addons non-xul rebuild: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Created blank page)
 
 
(8 intermediate revisions by the same user not shown)
Line 1: Line 1:
'''Revision History'''


{| class="wikitable" style="width:60%"
|-
! Date !! Version !! Author !! Description
|-
| 02/14/2018 || 1.0 || Cosmin Badescu || Created first draft
|-
|}
= Overview =
*Building new design and functionality in HTML will allow us to iterate faster, without the burden of dealing with XUL quirks and limitations.
*It would help to avoid investing deeper in a technology that is deprecated.
== Purpose ==
This document purports to detail the test approach for the improvements of Local Storage API  and includes Entry/Exit criteria, Scope for testing, links to test cases etc
=== Entry Criteria ===
* QA has access to all the PRDs, mocks and related documents
* The feature has landed on Nightly
* AMO parts has landed on dev
=== Exit Criteria ===
* All the bugs against the feature have been triaged
* All the P1/P2 bugs have been fixed
* All the resolved bugs have been verified by QA
* The find/fixed rate is going down over a predefined period of time
=== Acceptance Criteria ===
This section broadly outlines when the product is ready to ship
* QA has signed off
* All info is localized at least for a pre-defined set of locales
* All the necessary PR/blogposts have been sent out
== Scope ==
This section describes what parts of the feature will be tested and what parts won't be.
===what's in scope?===
* The XUL interface from about:addons converted to HTML which will iterate faster.
===what's out of scope?===
* Security testing
== Ownership ==
Dev Lead: [mailto:mstriemer@mozilla.com Mark Striemer]; irc nick :mstriemer <br />
QA Manager: [mailto:kraj@mozilla.com Krupa Raj]; irc nick :krupa<br />
QA Lead: [mailto:vcarciu@softvision.ro Victor Carciu]; irc nick :victorc<br />
Webextensions QA: [mailto:cosmin.badescu@softvision.ro Cosmin Badescu]; irc nick :CosminB<br />
= Requirements for testing =
== Environments ==
OSes covered: Windows, Mac OS X, Linux
== Channel dependent settings (configs) and environment setups ==
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
=== Nightly ===
<div class="mw-collapsible-content">
*Enable extensions.htmlaboutaddons.enabled from about:config
</div>
=== Beta ===
<div class="mw-collapsible-content">
*Enable extensions.htmlaboutaddons.enabled from about:config
</div>
=== Release ===
<div class="mw-collapsible-content">
The feature is enabled by default
</div>
</div>
= Test Strategy =
== Test Objectives ==
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.
This could be documented in bullet form or in a table similar to the one below.
{| class="wikitable"
|-
! Ref !! Function !! Test Objective !! Test Type !! Owners
|-
| TO-1 || Installing from AMO || To verify that the extension uses the API correctly || Manual || Add-ons QA Team
|-
| TO-2 || Installing from local files || To verify that the extension uses the API correctly || Manual || Add-ons QA Team
|-
| TO-3 || Add-on updates || To verify that the extension uses the API correctly || Manual || Add-ons QA Team
|-
| TO-4 || Sideloading || To verify that the extension uses the API correctly || Manual || Add-ons QA Team
|}
== Builds ==
This section should contain links for builds with the feature -
* [http://archive.mozilla.org/pub/mobile/nightly/latest-date-android-api-16/ Link] for Nightly builds
* [http://archive.mozilla.org/pub/mobile/candidates/ Link] for Beta builds
* [http://archive.mozilla.org/pub/mobile/releases/ Link] for Release builds
== Test Execution Schedule ==
The following table identifies the anticipated testing period available for test execution.
{| class="wikitable" style="width:60%"
|-
! Project phase !! Start Date !! End Date
|-
| Start project
|style="text-align:center;" |  ||
|-
| Study documentation/specs received from developers
|style="text-align:center;" |  ||
|-
| QA - Test plan creation
|style="text-align:center;" | 02-14-2018 ||
|-
| QA - Test cases/Env preparation
|style="text-align:center;" | 02-13-2018 || N/A
|-
| QA - Nightly Testing
|style="text-align:center;" |  ||
|-
| QA - Beta Testing
|style="text-align:center;" |  ||
|-
| Release Date
|style="text-align:center;" | ||
|}
== Testing Tools ==
Detail the tools to be used for testing, for example see the following table:
{| class="wikitable" style="width:50%"
|-
! Process !! Tool
|-
| Test plan creation || Mozilla wiki
|-
| Test case creation || [ Docs] / [https://testrail.stage.mozaws.net/index.php?/suites/view/3377&group_by=cases:section_id&group_order=asc TestRail]
|-
| Test case execution || [ Docs] / [ TestRail]
|-
| Bugs management || Bugzilla / Github
|}
= Status =
== Overview ==
Track the dates and build number where feature was released to Nightly
Track the dates and build number where feature was merged to Release/Beta
== Risk analysis ==
Identify the high-risk assumptions
Identify existing bugs on the feature with high risk
Identify if other areas are affected by the fix
= References =
* List and links for specs
  PRD - []
  Install flow - N/A
 
* {{Bug|1505924}} - [meta] New about:addons interface
<bugzilla>
    {
        "blocks":[1505924],
        "include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"
    }
</bugzilla>
= Testcases =
== Overview ==
Summary of testing scenarios
== Test Areas ==
{| class="wikitable" style="width:80%"
|-
! Test Areas !! Covered !! Details
|-
| '''Installing from AMO'''
|style="text-align:center;" | ||
|-
| '''Installing from local files'''
|style="text-align:center;" |  ||
|-
| '''Installing from thirdparty'''
|style="text-align:center;" |  ||
|-
| '''Add-on updates'''
|style="text-align:center;" |  ||
|-
| '''Sideloading'''
||  ||
|-
| '''Other''' 
|style="text-align:center;" |  ||
|-
|}
== Test suite ==
* Link for the [ Initial test planning]
* Link for the [ Google doc tests]
* Link for the [https://testrail.stage.mozaws.net/index.php?/suites/view/3377&group_by=cases:section_id&group_order=asc about:addons_non-xul_rebuild]
= Bug Work =
Tracking bug - []
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
====== Bug fix verification ======
<div class="mw-collapsible-content">
<font color="green">'''[Verified]'''</font> [ Bug xxxxxxx] - Display permissions prompt for webextensions installed using mozAddonManager
:↳ <small>'''2017-01-10'''</small>: verified fixed on 53.0a1 across platforms
<font color="green">'''[Verified]'''</font> [ Bug xxxxxxx] - Prompt users with permissions for third-party webextensions installs
:↳ <small>'''2015-04-21'''</small>: verified fixed on 53.0a1 across platforms
</div></div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
====== Logged bugs ======
<div class="mw-collapsible-content">
[ Bug xxxxxxx] - Misaligned icon and webextension name in permissions doorhanger<br/>
</div>
</div>
= Sign off =
== Criteria ==
Check list
* All test cases should be executed
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
== Results ==
'''Nightly testing'''<br />
List of OSes that will be covered by testing<br />
*Link for the tests run
** Full Test suite, use template from []
'''Merge to Beta Sign-off'''
List of OSes that will be covered by testing<br />
*Link for the tests run
** Full Test suite
== Checklist ==
{| class="wikitable" style="width:60%"
|-
! Exit Criteria !! Status !! Notes/Details
|-
|  Testing Prerequisites (specs, use cases)
| style="text-align:center;" | 
| style="text-align:center;" |
|-
|  Testing Infrastructure setup
|style="text-align:center;" |  ||
|-
|  Test Plan Creation
| style="text-align:center;" | 02-14-2019  ||
|-
|  Test Cases Creation
|style="text-align:center;" |  02-13-2019 || N/A
|-
|  Full Functional Tests Execution
|style="text-align:center;" | 
|style="text-align:center;" | 
|-
|  Automation Coverage ||
|style="text-align:center;" |
|-
|  Performance Testing
|style="text-align:center;" |  ||
|-
|  All Defects Logged || ||
|-
|  Critical/Blockers Fixed and Verified || ||
|-
|  Metrics/Telemetry||
|style="text-align:center;" |
|-
|  QA Signoff  - Nightly Release||
|style="text-align:center;" | Email to be sent
|-
|  QA Beta - Full Testing
|style="text-align:center;" |  ||
|-
|  QA Signoff  - Beta Release ||
|style="text-align:center;" | Email to be sent
|}

Latest revision as of 06:58, 8 May 2019

Revision History

Date Version Author Description
02/14/2018 1.0 Cosmin Badescu Created first draft


Overview

  • Building new design and functionality in HTML will allow us to iterate faster, without the burden of dealing with XUL quirks and limitations.
  • It would help to avoid investing deeper in a technology that is deprecated.

Purpose

This document purports to detail the test approach for the improvements of Local Storage API and includes Entry/Exit criteria, Scope for testing, links to test cases etc

Entry Criteria

  • QA has access to all the PRDs, mocks and related documents
  • The feature has landed on Nightly
  • AMO parts has landed on dev

Exit Criteria

  • All the bugs against the feature have been triaged
  • All the P1/P2 bugs have been fixed
  • All the resolved bugs have been verified by QA
  • The find/fixed rate is going down over a predefined period of time

Acceptance Criteria

This section broadly outlines when the product is ready to ship

  • QA has signed off
  • All info is localized at least for a pre-defined set of locales
  • All the necessary PR/blogposts have been sent out

Scope

This section describes what parts of the feature will be tested and what parts won't be.

what's in scope?

  • The XUL interface from about:addons converted to HTML which will iterate faster.

what's out of scope?

  • Security testing

Ownership

Dev Lead: Mark Striemer; irc nick :mstriemer
QA Manager: Krupa Raj; irc nick :krupa
QA Lead: Victor Carciu; irc nick :victorc
Webextensions QA: Cosmin Badescu; irc nick :CosminB


Requirements for testing

Environments

OSes covered: Windows, Mac OS X, Linux

Channel dependent settings (configs) and environment setups

Nightly

  • Enable extensions.htmlaboutaddons.enabled from about:config

Beta

  • Enable extensions.htmlaboutaddons.enabled from about:config

Release

The feature is enabled by default

Test Strategy

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref Function Test Objective Test Type Owners
TO-1 Installing from AMO To verify that the extension uses the API correctly Manual Add-ons QA Team
TO-2 Installing from local files To verify that the extension uses the API correctly Manual Add-ons QA Team
TO-3 Add-on updates To verify that the extension uses the API correctly Manual Add-ons QA Team
TO-4 Sideloading To verify that the extension uses the API correctly Manual Add-ons QA Team

Builds

This section should contain links for builds with the feature -

  • Link for Nightly builds
  • Link for Beta builds
  • Link for Release builds

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project
Study documentation/specs received from developers
QA - Test plan creation 02-14-2018
QA - Test cases/Env preparation 02-13-2018 N/A
QA - Nightly Testing
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation [ Docs] / TestRail
Test case execution [ Docs] / [ TestRail]
Bugs management Bugzilla / Github

Status

Overview

Track the dates and build number where feature was released to Nightly
Track the dates and build number where feature was merged to Release/Beta

Risk analysis

Identify the high-risk assumptions
Identify existing bugs on the feature with high risk
Identify if other areas are affected by the fix

References

* List and links for specs
  PRD - []
  Install flow - N/A
  



* bug 1505924 - [meta] New about:addons interface
Full Query
ID Priority Component Assigned to Summary Status Resolution Target milestone
1499500 P3 Add-ons Manager William Durand [:willdurand] Find in page is not possible in Add-ons manager RESOLVED FIXED 112 Branch
1514316 P2 Add-ons Manager Mark Striemer [:mstriemer] Setup a new about:addons behind a pref VERIFIED FIXED mozilla67
1525090 P2 Add-ons Manager Mark Striemer [:mstriemer] Use AddonListeners for HTML about:addons views VERIFIED FIXED mozilla67
1530530 P2 Add-ons Manager Show something when there are no add-ons installed RESOLVED DUPLICATE ---
1531967 P3 Add-ons Manager Use toggles instead of radio buttons for add-on options NEW ---
1533795 P1 Add-ons Manager Mark Striemer [:mstriemer] [meta] about:addons interface changes [phase 1] RESOLVED FIXED ---
1540173 -- Add-ons Manager [meta] change "Get Add-ons" (discopane) to be a proper part of about:addons rather than a framed page from AMO RESOLVED FIXED ---
1544111 P3 Themes Tim Nguyen :ntim Fix HTML about:addons to work with dark in-content pages VERIFIED FIXED mozilla68
1546711 P3 Add-ons Manager [about:addons] When aboutaddons.html frame is reloaded, it stays blank until the parent about:addons is reloaded RESOLVED FIXED ---
1549450 P2 Add-ons Manager Mark Striemer [:mstriemer] Cleanup inline options browser hacks RESOLVED FIXED ---
1550938 -- Add-ons Manager [about:addons] New HTML add-ons manager shows info about extensions in private windows for dictionaries and plugins RESOLVED DUPLICATE ---
1553608 P3 Add-ons Manager Use equal instead of is in new mochitests NEW ---
1555012 P1 Add-ons Manager Rob Wu [:robwu] Enable HTML about:addons by default VERIFIED FIXED mozilla69
1555372 P3 Add-ons Manager Remove eslint comments in toolkit/mozapps/extensions NEW ---
1556776 -- Add-ons Manager [meta] Convert about:addons entirely to HTML [phase 2] RESOLVED FIXED ---
1558073 -- Add-ons Manager Center content in Add-ons page (like other about: pages) RESOLVED INVALID ---
1558823 P3 Add-ons Manager Oriol Brufau [:Oriol] Addons without icon should show something (default addon-icon) VERIFIED FIXED mozilla70
1558982 P2 Add-ons Manager [meta] Remove XUL about:addons RESOLVED FIXED ---
1559788 -- Add-ons Manager Always show the control buttons in about:addons RESOLVED WONTFIX ---
1560759 -- Add-ons Manager Handle addons that have been removed from AMO more gracefully RESOLVED INVALID ---
1562425 P3 Add-ons Manager Emilio Cobos Álvarez (:emilio) [RTL][about:addons] Some icons appear on the wrong side of their container VERIFIED FIXED mozilla70
1564305 P3 Add-ons Manager Add-on recommendation listing name should be linked; not obvious that author link links to the add-on RESOLVED DUPLICATE ---
1564967 -- Add-ons Manager Tom S. (please needinfo tschuster) Context menu entry "Options" and "More options" is confusing VERIFIED FIXED mozilla70
1564991 P3 Add-ons Manager about:addons should have a tabbed UI to make thing easier to find (eg recommendations) NEW ---
1565057 -- Add-ons Manager Bring back some features hidden behind meatball menu for commonly used tasks RESOLVED DUPLICATE ---
1565122 -- Add-ons Manager Missing Options/Preferences Panel with new about:addons UI RESOLVED DUPLICATE ---
1565596 P3 Add-ons Manager [about:addons] Truncate long homepage URLs NEW ---
1565987 P3 Add-ons Manager Context menu ‘Options/Preferences’ entry is displayed only after page refresh NEW ---
1566021 P3 Add-ons Manager Navigating via ‘Tab’ key will get the user stuck in a loop in the details page NEW ---
1610084 P5 Add-ons Manager Page Down and Page Up not immediately usable on about:addons in FF 72 NEW ---

30 Total; 8 Open (26.67%); 15 Resolved (50%); 7 Verified (23.33%);


Testcases

Overview

Summary of testing scenarios

Test Areas

Test Areas Covered Details
Installing from AMO
Installing from local files
Installing from thirdparty
Add-on updates
Sideloading
Other

Test suite

Bug Work

Tracking bug - []

Bug fix verification

[Verified] [ Bug xxxxxxx] - Display permissions prompt for webextensions installed using mozAddonManager

↳ 2017-01-10: verified fixed on 53.0a1 across platforms

[Verified] [ Bug xxxxxxx] - Prompt users with permissions for third-party webextensions installs

↳ 2015-04-21: verified fixed on 53.0a1 across platforms
Logged bugs

[ Bug xxxxxxx] - Misaligned icon and webextension name in permissions doorhanger

Sign off

Criteria

Check list

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite, use template from []


Merge to Beta Sign-off List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation 02-14-2019
Test Cases Creation 02-13-2019 N/A
Full Functional Tests Execution
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
QA Signoff - Nightly Release Email to be sent
QA Beta - Full Testing
QA Signoff - Beta Release Email to be sent
Retrieved from "https://wiki.allizom.org/index.php?title=Add-ons/QA/Testplan/about:addons_non-xul_rebuild&oldid=1211970"