Add-ons/QA/Testplan/Improve Storage Local API: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
Line 5: Line 5:
! Date !! Version !! Author !! Description  
! Date !! Version !! Author !! Description  
|-
|-
| 05/22/2018 || 1.0 || Cosmin Badescu || Created first draft
| 07/16/2018 || 1.0 || Cosmin Badescu || Created first draft
|-
|-
|}
|}
Line 11: Line 11:


= Overview =
= Overview =
The purpose of this feature is to check that the migration from storage.local JSONFile backend to storage.local IndexedDB backend works correctly.
TBD.


== Purpose ==
== Purpose ==
This document purports to detail the test approach for the improvements of Local Storage API and includes Entry/Exit criteria, Scope for testing, links to test cases etc
This document purports to detail the test approach for the improvements of Context Menu Updates and includes Entry/Exit criteria, Scope for testing, links to test cases etc


=== Entry Criteria ===
=== Entry Criteria ===
Line 38: Line 38:


===what's in scope?===
===what's in scope?===
* The behavior of the storage.local JSONFile backend and storage.local IndexedDB backend when the extensions use them.
* TBD.


===what's out of scope?===
===what's out of scope?===
Line 44: Line 44:


== Ownership ==
== Ownership ==
Dev Lead: [mailto:lgreco@mozilla.com Luca Greco]; irc nick :rpl <br />
Dev Lead: [dev]; irc nick :dev <br />
QA Manager: [mailto:kraj@mozilla.com Krupa Raj]; irc nick :krupa<br />  
QA Manager: [mailto:kraj@mozilla.com Krupa Raj]; irc nick :krupa<br />  
QA Lead: [mailto:vcarciu@softvision.ro Victor Carciu]; irc nick :victorc<br />
QA Lead: [mailto:vcarciu@softvision.ro Victor Carciu]; irc nick :victorc<br />
Line 59: Line 59:
=== Nightly ===
=== Nightly ===
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
*Enable extensions.webextensions.ExtensionStorageIDB.enabled from about:config
*TBD
</div>
</div>


=== Beta ===
=== Beta ===
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
*Enable extensions.webextensions.ExtensionStorageIDB.enabled from about:config
*TBD
</div>
</div>


Line 111: Line 111:
|-
|-
| QA - Test plan creation  
| QA - Test plan creation  
|style="text-align:center;" | 05-22-2018 ||  
|style="text-align:center;" | 07-16-2018 ||  
|-
|-
| QA - Test cases/Env preparation  
| QA - Test cases/Env preparation  
|style="text-align:center;" | 05-21-2018 || N/A
|style="text-align:center;" | || N/A
|-
|-
| QA - Nightly Testing  
| QA - Nightly Testing  
Line 153: Line 153:
= References =
= References =
  * List and links for specs
  * List and links for specs
   PRD - TBD
   PRD - [https://docs.google.com/document/d/1dVMb2uSgP7vgwXHatyOXpCPmCBK5hF-szFZ3Tzb5ItA/edit link]
  Install flow - [https://docs.google.com/document/d/1IDNHoneiARuw8PT6KIQGp03WxFmaQYijpluZA0C-E7I/edit Link]
  Install flow - TBD
    
    




  * {{Bug|1406181}} - Use indexedDB as the backend for storage.local (pref Off)
  * {{Bug|1466876}} - [META] context menu updates
<bugzilla>
<bugzilla>
     {
     {
         "blocks":[1406181],
         "blocks":[1466876],
         "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
         "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
     }
     }
</bugzilla>
</bugzilla>


  * {{Bug|1474562}} - Enable ExtensionStorageIDB backend on Nightly (pref ON)
  * {{Bug|}} -  
<bugzilla>
<bugzilla>
     {
     {
         "blocks":[1474562],
         "blocks":[],
         "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
         "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
     }
     }
Line 205: Line 205:
* Link for the [ Initial test planning]
* Link for the [ Initial test planning]
* Link for the [ Google doc tests]
* Link for the [ Google doc tests]
* Link for the [https://testrail.stage.mozaws.net/index.php?/suites/view/1735&group_by=cases:section_id&group_order=asc TestRail tests]
* Link for the [ TestRail tests]


= Bug Work =
= Bug Work =
Line 262: Line 262:
|-
|-
|  Test Plan Creation  
|  Test Plan Creation  
| style="text-align:center;" | 05-22-2018  ||  
| style="text-align:center;" | 07-16-2018  ||  
|-
|-
|  Test Cases Creation  
|  Test Cases Creation  
|style="text-align:center;" |  05-21-2018 || N/A
|style="text-align:center;" |  || N/A
|-
|-
|  Full Functional Tests Execution  
|  Full Functional Tests Execution  

Revision as of 12:19, 16 July 2018

Revision History

Date Version Author Description
07/16/2018 1.0 Cosmin Badescu Created first draft


Overview

TBD.

Purpose

This document purports to detail the test approach for the improvements of Context Menu Updates and includes Entry/Exit criteria, Scope for testing, links to test cases etc

Entry Criteria

  • QA has access to all the PRDs, mocks and related documents
  • The feature has landed on Nightly
  • AMO parts has landed on dev

Exit Criteria

  • All the bugs against the feature have been triaged
  • All the P1/P2 bugs have been fixed
  • All the resolved bugs have been verified by QA
  • The find/fixed rate is going down over a predefined period of time

Acceptance Criteria

This section broadly outlines when the product is ready to ship

  • QA has signed off
  • All the required Telemetry is in place
  • All info is localized at least for a pre-defined set of locales
  • All the necessary PR/blogposts have been sent out

Scope

This section describes what parts of the feature will be tested and what parts won't be.

what's in scope?

  • TBD.

what's out of scope?

  • Security testing

Ownership

Dev Lead: [dev]; irc nick :dev
QA Manager: Krupa Raj; irc nick :krupa
QA Lead: Victor Carciu; irc nick :victorc
Webextensions QA: Cosmin Badescu; irc nick :CosminB


Requirements for testing

Environments

OSes covered: Windows, Mac OS X, Linux

Channel dependent settings (configs) and environment setups

Nightly

  • TBD

Beta

  • TBD

Release

The feature is enabled by default

Test Strategy

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref Function Test Objective Test Type Owners
TO-1 Installing from AMO To verify that the extension uses the API correctly Manual Add-ons QA Team
TO-2 Installing from local files To verify that the extension uses the API correctly Manual Add-ons QA Team
TO-3 Add-on updates To verify that the extension uses the API correctly Manual Add-ons QA Team
TO-4 Sideloading To verify that the extension uses the API correctly Manual Add-ons QA Team

Builds

This section should contain links for builds with the feature -

  • Link for Nightly builds
  • Link for Beta builds
  • Link for Release builds

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project
Study documentation/specs received from developers
QA - Test plan creation 07-16-2018
QA - Test cases/Env preparation N/A
QA - Nightly Testing
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation [ Docs] / [ TestRail]
Test case execution [ Docs] / [ TestRail]
Bugs management Bugzilla / Github

Status

Overview

Track the dates and build number where feature was released to Nightly
Track the dates and build number where feature was merged to Release/Beta

Risk analysis

Identify the high-risk assumptions
Identify existing bugs on the feature with high risk
Identify if other areas are affected by the fix

References

* List and links for specs
  PRD - link
  Install flow - TBD
  



* bug 1466876 - [META] context menu updates
Full Query
ID Priority Component Assigned to Summary Status Target milestone
1280347 P2 General Rob Wu [:robwu] Add ability to provide custom HTML elements working as alias of existing Firefox UI items, especially tabs VERIFIED mozilla64
1280370 P3 Untriaged Rob Wu [:robwu] contextMenus do not support other protocols (ex. 'magnet:*', 'acestream:*', 'sop:*') VERIFIED mozilla63
1320462 P2 General Rob Wu [:robwu] Add ability to set access key to context menu item VERIFIED mozilla63
1325814 P3 General Rob Wu [:robwu] contextMenus.OnClickData should provide selector of the clicked element VERIFIED mozilla63
1366702 P3 Frontend Rob Wu [:robwu] Callback for contextMenus.create is not called when a failure occurs VERIFIED mozilla63
1367160 P2 Frontend Rob Wu [:robwu] Consider removing default context menu items in extension frames RESOLVED mozilla64
1398453 P3 Frontend Rob Wu [:robwu] Context menu info srcUrl is blank in youtube videos RESOLVED ---
1398542 P5 General Rob Wu [:robwu] Need test coverage for video and audio types in contextMenus / menus API RESOLVED mozilla63
1416839 P2 Frontend Rob Wu [:robwu] Restrict context menus from showing up in the sidebar RESOLVED mozilla64
1446956 P2 Frontend Rob Wu [:robwu] activeTab permission not granted for tab on which the contextmenu is used VERIFIED mozilla63
1482529 P3 Frontend Rob Wu [:robwu] Implement "visible" parameter in menus.create and menus.update VERIFIED mozilla63
1488524 P3 Frontend Support fill="context-fill" for Mozilla WebExtension Context Menu Icons RESOLVED ---
1492969 P3 Frontend Icon overlaps with radio/checkbox element on menu items on Linux NEW ---
1498896 -- Frontend Rob Wu [:robwu] Always use documentUrlPatterns to match the document URL in extension views, even when menus.overrideContext is used VERIFIED mozilla65

14 Total; 1 Open (7.14%); 5 Resolved (35.71%); 8 Verified (57.14%); 


* bug  -

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 400 [1] => Bad Request ) ) 1


Testcases

Overview

Summary of testing scenarios

Test Areas

Test Areas Covered Details
Installing from AMO
Installing from local files
Installing from thirdparty
Add-on updates
Sideloading
Other

Test suite

  • Link for the [ Initial test planning]
  • Link for the [ Google doc tests]
  • Link for the [ TestRail tests]

Bug Work

Tracking bug - []

Bug fix verification

[Verified] [ Bug xxxxxxx] - Display permissions prompt for webextensions installed using mozAddonManager

↳ 2017-01-10: verified fixed on 53.0a1 across platforms

[Verified] [ Bug xxxxxxx] - Prompt users with permissions for third-party webextensions installs

↳ 2015-04-21: verified fixed on 53.0a1 across platforms
Logged bugs

[ Bug xxxxxxx] - Misaligned icon and webextension name in permissions doorhanger

Sign off

Criteria

Check list

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite, use template from []


Merge to Beta Sign-off List of OSes that will be covered by testing

  • Link for the tests run
    • Full Test suite

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases)
Testing Infrastructure setup
Test Plan Creation 07-16-2018
Test Cases Creation N/A
Full Functional Tests Execution
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Metrics/Telemetry
QA Signoff - Nightly Release Email to be sent
QA Beta - Full Testing
QA Signoff - Beta Release Email to be sent
Retrieved from "https://wiki.allizom.org/index.php?title=Add-ons/QA/Testplan/Improve_Storage_Local_API&oldid=1197286"