|
|
Line 1: |
Line 1: |
| '''Revision History'''
| | Test |
| | |
| {| class="wikitable" style="width:60%"
| |
| |-
| |
| ! Date !! Version !! Author !! Description
| |
| |-
| |
| | 07/19/2017 || 1.0 || Cosmin Badescu || Created first draft
| |
| |-
| |
| |}
| |
| | |
| | |
| = Overview =
| |
| | |
| | |
| == Purpose ==
| |
| | |
| | |
| === Entry Criteria ===
| |
| * QA has access to all the PRDs, mocks and related documents
| |
| * The feature has landed on Nightly
| |
| | |
| === Exit Criteria ===
| |
| * All the bugs against the feature have been triaged
| |
| * All the P1/P2 bugs have been fixed
| |
| * All the resolved bugs have been verified by QA
| |
| * The find/fixed rate is going down over a predefined period of time
| |
| | |
| === Acceptance Criteria ===
| |
| This section broadly outlines when the product is ready to ship
| |
| * QA has signed off
| |
| * All the required Telemetry is in place
| |
| * All info is localized at least for a pre-defined set of locales
| |
| * All the necessary PR/blogposts have been sent out
| |
| | |
| == Scope ==
| |
| This section describes what parts of the feature will be tested and what parts won't be.
| |
| | |
| ===what's in scope?===
| |
| * Webextensions installation with e10s enabled
| |
| * Webextensions running in their own process
| |
| * Regression testing for the top 20 webextensions
| |
| * Regression testing for webextensions examples
| |
| * Regression testing for old XUL add-ons
| |
| * Webextension crashing
| |
| * Add-ons behavior after a Firefox crash
| |
| | |
| ===what's out of scope?===
| |
| * Security testing
| |
| * Performance testing
| |
| | |
| == Ownership ==
| |
| Dev Lead: [mailto:kmaglione@mozilla.com> Kris Maglione]; irc nick <br />
| |
| QA Manager: [mailto:kraj@mozilla.com Krupa Raj]; irc nick :krupa<br />
| |
| QA Lead: [mailto:vcarciu@softvision.ro Victor Carciu]; irc nick :victorc<br />
| |
| Webextensions QA: [mailto:vasilica.mihasca@softvision.ro Vasilica Mihasca]; irc nick :vasilica_mihasca<br />
| |
| Add-ons QA: [mailto:valentina.peleskei@softvision.ro Valentina Peleski]; irc nick :ValentinaV<br />
| |
| | |
| = Requirements for testing =
| |
| == Environments ==
| |
| OSed covered: Windows, Mac OS X, Linux
| |
| | |
| == Channel dependent settings (configs) and environment setups ==
| |
| <div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
| |
| | |
| === Nightly ===
| |
| <div class="mw-collapsible-content">
| |
| * Disable check for signing if testing on AMO dev or stage
| |
| * Webextension process crashing:
| |
| ** Set "extensions.webextensions.remote” to true
| |
| ** Insert the following line in Browser Console input filed: Cu.import("resource://gre/modules/Extension.jsm").GlobalManager.extensionMap.values().next().value.parentMessageManager.loadProcessScript("data:,Components.utils.import('resource://gre/modules/ctypes.jsm').ctypes.cast(ctypes.uintptr_t(1), ctypes.int.ptr).contents", false)
| |
| </div>
| |
| | |
| === Beta ===
| |
| <div class="mw-collapsible-content">
| |
| * [Unbranded build] Disable check for signing if testing on AMO dev or stage
| |
| | |
| </div>
| |
| | |
| === Post Beta / Release ===
| |
| <div class="mw-collapsible-content">
| |
| The feature is enabled by default.
| |
| </div>
| |
| </div>
| |
| | |
| == Test Objectives ==
| |
| This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.
| |
| This could be documented in bullet form or in a table similar to the one below.
| |
| | |
| {| class="wikitable"
| |
| |-
| |
| ! Ref !! Function !! Test Objective !! Test Type !! Owners
| |
| |-
| |
| | TO-1 || Installing with e10s enabled || To verify that installation process runs without any issue || Manual || Add-ons QA Team
| |
| |-
| |
| | TO-2 || Webextensions running in their own process || To verify that webextensions run in separate process || Manual || Add-ons QA Team
| |
| |-
| |
| | TO-3 || Regression testing for the top 20 webextensions || To verify that the most popular webextensions works without issues || Manual || Add-ons QA Team
| |
| |-
| |
| | TO-4 || Regression testing for webextensions example master || To verify that the webextensions works without issues || Manual || Add-ons QA Team
| |
| |-
| |
| | TO-5 || Webextensions testing in Private Window|| To verify the webextension behavior in private winodow || Manual || Add-ons QA Team
| |
| |-
| |
| | TO-6 || Webextension crashing || To verify that crashing the webextension process does not cause a Firefox crash || Manual || Add-ons QA Team
| |
| |-
| |
| | TO-7 || Firefox crashing || To verify that Firefox will load all the add-ons properly without any loss of saved add-on data || Manual || Add-ons QA Team
| |
| |-
| |
| | TO-8 || Regression testing on XUL add-ons || To verify that the old XUL add-ons works without any issue || Manual || Add-ons QA Team
| |
| |}
| |
| | |
| == Test Strategy ==
| |
| Here are the main ways in which we will ascertain that OOP functions as expected
| |
| * Test all the webextensions listed at https://github.com/mdn/webextensions-examples and ensure there are no regressions
| |
| * Check webextensions which are using APIs which are most affected by OOP. These APIs include -
| |
| ** context Menus
| |
| ** devtools
| |
| ** omnibox
| |
| ** tabs
| |
| ** identity
| |
| ** permissions
| |
| * Check that a firefox doesn't crash if a webextension crashes. See [https://bugzilla.mozilla.org/show_bug.cgi?id=1353959 Bug 1353959] for more details
| |
| * Check that extensions work as expected after a firefox crash
| |
| | |
| webRequest
| |
| == Builds ==
| |
| This section should contain links for builds with the feature -
| |
| * [http://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/ Link] for Nightly builds
| |
| * [https://archive.mozilla.org/pub/firefox/candidates/ Link] for Beta builds
| |
| * Beta Unbranded builds
| |
| ** [https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-win64-add-on-devel/ Windows Link]
| |
| ** [https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-linux64-add-on-devel/ Linux Link]
| |
| ** [https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-macosx64-add-on-devel/ Mac Link]
| |
| | |
| == Test Execution Schedule ==
| |
| The following table identifies the anticipated testing period available for test execution.
| |
| {| class="wikitable" style="width:60%"
| |
| |-
| |
| ! Project phase !! Start Date !! End Date
| |
| |-
| |
| | Start project
| |
| |style="text-align:center;" | ||
| |
| |-
| |
| | Study documentation/specs received from developers
| |
| |style="text-align:center;" | ||
| |
| |-
| |
| | QA - Test plan creation
| |
| |style="text-align:center;" | 02-03-2017 ||
| |
| |-
| |
| | QA - Test cases/Env preparation
| |
| |style="text-align:center;" | ||
| |
| |-
| |
| | QA - Nightly Testing
| |
| |style="text-align:center;" | 03-22-2017||
| |
| |-
| |
| | QA - Beta Testing
| |
| |style="text-align:center;" | ||
| |
| |-
| |
| | Release Date
| |
| |style="text-align:center;" | ||
| |
| |}
| |
| | |
| == Testing Tools ==
| |
| Detail the tools to be used for testing, for example see the following table:
| |
| {| class="wikitable" style="width:50%"
| |
| |-
| |
| ! Process !! Tool
| |
| |-
| |
| | Test plan creation || Mozilla wiki
| |
| |-
| |
| | Test case creation || [ TestRail] / [Google docs] / [https://public.etherpad-mozilla.org/p/OOP-WebExtension-Testing etherpad]
| |
| |-
| |
| | Test case execution || [TestRail]
| |
| |-
| |
| | Bugs management || Bugzilla / Github
| |
| |}
| |
| | |
| = Status =
| |
| == Overview ==
| |
| Track the dates and build number where feature was released to Nightly
| |
| Track the dates and build number where feature was merged to Aurora
| |
| Track the dates and build number where feature was merged to Release/Beta
| |
| | |
| == Risk analysis ==
| |
| * The test strategy recommended by devs is based on spot-checking and regression testing.
| |
| ** Mitigation stragety - QA will test all webextensions with the most number of users to ensure that the most impactful add-ons are covered.
| |
| * Webextensions will have their own process but Content scripts will still run in the content process
| |
| ** Mitigation stragety - QA will test with webextensions which run content scripts to make sure there is degradation of behavior
| |
| * Product is yet to define what is the acceptable dip in performance for when webextensions run in their process
| |
| | |
| = References =
| |
| * List and links for specs
| |
| PRD -
| |
| * {{Bug|1185785}} - (webext-android) [tracking] Extension API for Android
| |
| <bugzilla>
| |
| {
| |
| "blocks":[1185785],
| |
| "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
| |
| }
| |
| </bugzilla>
| |
| | |
| | |
| = Bug Work =
| |
| Tracking bug - [https://bugzilla.mozilla.org/show_bug.cgi?id=1185785 bug 1185785]
| |
| | |
| <div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
| |
| | |
| ====== Bug fix verification ======
| |
| <div class="mw-collapsible-content">
| |
| </div></div>
| |
| | |
| <div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
| |
| | |
| ====== Logged bugs ======
| |
| <div class="mw-collapsible-content">
| |
| | |
| </div>
| |
| </div>
| |
| | |
| = Sign off =
| |
| == Criteria ==
| |
| Check list
| |
| * All test cases should be executed
| |
| * Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
| |
| * All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
| |
| | |
| == Results ==
| |
| '''Nightly testing'''<br />
| |
| | |
| List of OSes that will be covered by testing<br />
| |
| *Link for the tests run
| |
| ** Daily Smoke, use template from [link]
| |
| ** Full Test suite, use template from [ link]
| |
| ** Regression Test suite, if needed/available
| |
| | |
| '''Merge to Aurora Sign-off'''
| |
| List of OSes that will be covered by testing<br />
| |
| *Link for the tests run
| |
| ** Full Test suite
| |
| | |
| == Checklist ==
| |
| {| class="wikitable" style="width:60%"
| |
| |-
| |
| ! Exit Criteria !! Status !! Notes/Details
| |
| |-
| |
| | Testing Prerequisites (specs, use cases)
| |
| | style="text-align:center;" |
| |
| | style="text-align:center;" |
| |
| |-
| |
| | Testing Infrastructure setup
| |
| |style="text-align:center;" | ||
| |
| |-
| |
| | Test Plan Creation
| |
| | style="text-align:center;" | ||
| |
| |-
| |
| | Test Cases Creation
| |
| |style="text-align:center;" | ||
| |
| |-
| |
| | Full Functional Tests Execution
| |
| |style="text-align:center;" |
| |
| |style="text-align:center;" |
| |
| |-
| |
| | Automation Coverage ||
| |
| |style="text-align:center;" |
| |
| |-
| |
| | Performance Testing
| |
| |style="text-align:center;" | ||
| |
| |-
| |
| | All Defects Logged || ||
| |
| |-
| |
| | Critical/Blockers Fixed and Verified || ||
| |
| |-
| |
| | Metrics/Telemetry||
| |
| |style="text-align:center;" |
| |
| |-
| |
| | QA Signoff - Nightly Release||
| |
| |style="text-align:center;" | Email to be sent
| |
| |-
| |
| | QA Aurora - Full Testing
| |
| |style="text-align:center;" | ||
| |
| |-
| |
| | QA Signoff - Aurora Release ||
| |
| |style="text-align:center;"| Email to be sent
| |
| |-
| |
| | QA Beta - Full Testing
| |
| |style="text-align:center;" | ||
| |
| |-
| |
| | QA Signoff - Beta Release ||
| |
| |style="text-align:center;" | Email to be sent
| |
| |}
| |