Add-ons/QA/Testplan/Auto-Approvals: Difference between revisions
ValentinaP (talk | contribs) |
ValentinaP (talk | contribs) |
||
| Line 110: | Line 110: | ||
= References = | = References = | ||
* PRD - https://docs.google.com/document/d/ | * PRD - https://docs.google.com/document/d/1VxvvzQQvfDOVCylSVlpfeXv0tDYRLZfIMvWOoraANGU/edit# | ||
* Tracker: | * Tracker: [https://github.com/mozilla/addons-server/issues/4533 4533] | ||
= Testcases = | = Testcases = | ||
Revision as of 08:02, 28 April 2017
Revision History
| Date | Version | Author | Description |
|---|---|---|---|
| 04/21/2017 | 1.0 | Valentina Virlics | Created first draft |
Overview
Currently, every add-on version listed on AMO is manually code reviewed by a human (called: reviewer). This process is meant to ensure a good user experience with add-ons. This manual review process requires significant human effort and this is why we need auto-approvals - a change that will be implemented in the review process, Reviewer Tools and submission to move to a hybrid system that includes human reviews as well as automatic reviews.
Purpose
Reducing delays in reviews (delays present due to decreased reviewer activity and increase submissions) by automatically reviewing some add-ons (that meet a list of criteria).
Entry Criteria
Current Status
- The feature was released.
Exit Criteria
- All related bugs triaged
- All blockers fixed
- All resolved bugs verified by QA
- Found-fixed bugs rate going down in time
Scope
what's in scope?
- Add a counter for approvals since last automatic review
- Get extensions auto-approved if they meet the following criteria:
- Have been manually approved once
- Are not flagged for admin review (this can happen automatically on submission in some cases, but is generally done manually by reviewers)
- Are WebExtensions
- Active Daily Users are under 10K (initially set to this value, but configurable).
- Are not triggering any of the following linter warnings:
- CSP
- Native Messaging
Not yet implemented:
- Auto-approve extensions that are not triggering content scripts warnings for all URLs at validation
what's out of scope?
- Human unreviewed extensions
- Embedded Webextensions
Ownership
Product Manager: Jorge Villalobos; irc nick :jorgev
QA Manager: Krupa Raj; irc nick :krupa
QA Lead: Victor Carciu; irc nick :victorc
Add-ons QA: Valentina Virlics; irc nick :ValentinaV
Requirements for testing
Environments
- Windows
- Mac OS
Servers
- Stage: https://addons.allizom.org/en-US/
- Dev: https://addons-dev.allizom.org/en-US/
- Production: https://addons.mozilla.org/en-US/
Channels
- Release
Test Strategy
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
| Project phase | Start Date | End Date |
|---|---|---|
| Start project | 31.01.2017 | |
| Study PRD/mocks received | 01.02.2017 | |
| QA - Test plan creation | 21.04.2017 | |
| QA - Test cases preparation | 21.04.2017 | |
| QA - Test cases execution | 24.04.2017 | |
| Release Date | 27.04.2017 (Q1 - 2017) |
Created an etherpad with more detailed scenarios - Auto-Approvals
Testing Tools
| Process | Tool |
|---|---|
| Test plan creation | Mozilla wiki |
| Test case creation | [TestRail] / [Google docs] |
| Test case execution | [TestRail] |
| Bugs management | Github |
References
* PRD - https://docs.google.com/document/d/1VxvvzQQvfDOVCylSVlpfeXv0tDYRLZfIMvWOoraANGU/edit# * Tracker: 4533
Testcases
Test Areas
- Submission of add-ons vs. submission of webextensions
Test suite
Full Test Suite: here
Bug Work
Bug fix verification
Logged bugs
Sign off
Criteria
Check list
- All test cases should be executed
- All blockers must be fixed and verified or have an agreed-upon timeline for being fixed