Add-ons/QA/Testplan/Android Webextension API
Revision History
Date | Version | Author | Description |
---|---|---|---|
07/17/2017 | 1.0 | Cosmin Badescu | Created first draft |
Overview
The implementation of the Android WebExtension API, is to ensure that the WebExtensions are functional on the Android platform.
Purpose
Entry Criteria
- QA has access to all the PRDs, mocks and related documents
- The feature has landed on Nightly
Exit Criteria
- All the bugs against the feature have been triaged
- All the P1/P2 bugs have been fixed
- All the resolved bugs have been verified by QA
- The find/fixed rate is going down over a predefined period of time
Acceptance Criteria
This section broadly outlines when the product is ready to ship
- QA has signed off
- All the required Telemetry is in place
- All info is localized at least for a pre-defined set of locales
- All the necessary PR/blogposts have been sent out
Scope
This section describes what parts of the feature will be tested and what parts won't be.
what's in scope?
- Webextensions installation with e10s enabled
- Webextensions running in their own process
- Regression testing for the top 20 WebExtensions
- Regression testing for WebExtensions examples
- Webextension crashing
- Add-ons behavior after a Firefox crash
- BrowserAction API used for creating buttons to the browser's toolbar
- PageAction API used for icon inside the browser's address bar that are clickable
what's out of scope?
- Security testing
- Performance testing
Ownership
Dev Lead: Matthew Wein; irc nick :mattw
QA Manager: Krupa Raj; irc nick :krupa
QA Lead: Victor Carciu; irc nick :victorc
Webextensions QA: Cosmin Badescu; irc nick :CosminB
Requirements for testing
Environments
OSed covered: Android
Channel dependent settings (configs) and environment setups
Test Objectives
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.
Ref | Function | Test Objective | Test Type | Owners |
---|---|---|---|---|
TO-1 | Installing with e10s enabled | To verify that installation process runs without any issue | Manual | Add-ons QA Team |
TO-2 | Webextensions running in their own process | To verify that webextensions run in separate process | Manual | Add-ons QA Team |
TO-3 | Regression testing for the top 20 webextensions | To verify that the most popular webextensions works without issues | Manual | Add-ons QA Team |
TO-4 | Regression testing for webextensions example master | To verify that the webextensions works without issues | Manual | Add-ons QA Team |
TO-5 | Webextensions testing in Private Window | To verify the webextension behavior in private winodow | Manual | Add-ons QA Team |
TO-6 | Webextension crashing | To verify that crashing the webextension process does not cause a Firefox crash | Manual | Add-ons QA Team |
TO-7 | Firefox crashing | To verify that Firefox will load all the add-ons properly without any loss of saved add-on data | Manual | Add-ons QA Team |
TO-8 | BrowserAction | To verify that Firefox will display and run a button in the main menu of the browser | Manual | Add-ons QA Team |
TO-9 | PageAction | To verify that Firefox will display an icon inside the browser's address bar, that listen for clicks or displays a pop-up that opens when the icon is clicked | Manual | Add-ons QA Team |
TO-10 | Options UI | To verify that Firefox will display in about:addons, on the detailed page of the WebExtension, an options page | Manual | Add-ons QA Team |
Test Strategy
Here are the main ways in which we will ascertain that Android Webextension API functions as expected
- Test the WebExtensions listed at https://addons-dev.allizom.org/en-US/android/ and ensure there are no regressions.
- Check that WebExtensions that are using Android Webextension API work as expected.
- Check that Firefox doesn't crash if a WebExtension crashes. See Bug 1353959 for more details.
- Check that WebExtensions work as expected after a firefox crash.
Builds
This section should contain links for builds with the feature:
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
Project phase | Start Date | End Date |
---|---|---|
Start project | 07-20-2015 | |
Study documentation/specs received from developers | ||
QA - Test plan creation | 07-17-2017 | |
QA - Test cases/Env preparation | 07-17-2017 | |
QA - Nightly Testing | 07-17-2017 | 07-26-2017 |
QA - Beta Testing | 09-14-2017 | |
Release Date | 09-28-2017 |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
Process | Tool |
---|---|
Test plan creation | Mozilla wiki |
Test case creation | TestRail / [Google docs] / |
Test case execution | TestRail |
Bugs management | Bugzilla |
Status
Overview
Track the dates and build number where feature was released to Nightly Track the dates and build number where feature was merged to Release/Beta
Risk analysis
- The test strategy recommended by devs is based on spot-checking and regression testing.
- Mitigation stragety - QA will test all webextensions with the most number of users to ensure that the most impactful WebExtensions are covered.
- Webextensions will have their own process but Content scripts will still run in the content process
- Mitigation stragety - QA will test with WebExtensions which run content scripts to make sure there is degradation of behavior
- Product is yet to define what is the acceptable dip in performance for when webextensions run in their process
References
* List and links for specs PRD - * bug 1185785 - (webext-android) [tracking] Extension API for Android
26 Total; 0 Open (0%); 21 Resolved (80.77%); 5 Verified (19.23%);
Bug Work
Tracking bug - bug 1185785
Bug fix verification
Logged bugs
Sign off
Criteria
Check list
- All test cases should be executed
- Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
Nightly testing
List of OSes that will be covered by testing
- Link for the tests run
- Daily Smoke, use template from [link]
- Full Test suite, use template from [ link]
- Regression Test suite, if needed/available
Merge to Aurora Sign-off
List of OSes that will be covered by testing
- Link for the tests run
- Full Test suite
Checklist
Exit Criteria | Status | Notes/Details |
---|---|---|
Testing Prerequisites (specs, use cases) | ||
Testing Infrastructure setup | ||
Test Plan Creation | ||
Test Cases Creation | ||
Full Functional Tests Execution | ||
Automation Coverage | ||
Performance Testing | ||
All Defects Logged | ||
Critical/Blockers Fixed and Verified | ||
Metrics/Telemetry | ||
QA Signoff - Nightly Release | Email to be sent | |
QA Beta - Full Testing | ||
QA Signoff - Beta Release | Email to be sent |