Confirmed users
1,595
edits
QA/Feature Testing: Difference between revisions
m
→Useful references: -
Andrei.vaida (talk | contribs) m (-) |
Andrei.vaida (talk | contribs) m (→Useful references: -) |
||
| (19 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
= Feature testing = | = Feature testing = | ||
{{note|The information below is also available as a Google Doc [https://docs.google.com/document/d/1PAF93b0PrFvUIK2FmxDNYMdTM2awPnWcanOIIX68ZG8/edit# here].}} | |||
== Purpose of this document == | == Purpose of this document == | ||
=== What this document is === | === What this document is === | ||
This document outlines the major milestones associated with the quality management of Firefox features. These milestones are centered around manual testing, evaluation and reporting. | This document outlines the major milestones associated with the quality management of Firefox features. These milestones are centered around manual testing, evaluation and reporting. | ||
{{ | {{warning|The process described below applies only for the Firefox features shipped based on the train model.}} | ||
=== What this document is not === | === What this document is not === | ||
| Line 11: | Line 10: | ||
* QA teams conducting automated testing | * QA teams conducting automated testing | ||
* QA teams working on off-the-train features | * QA teams working on off-the-train features | ||
{{ | {{warning|The process described below does not directly result in a final Go/NoGo decision for a Firefox feature.}} | ||
== Process overview == | == Process overview == | ||
=== Phase 1: Kickoff and Test Plan sign off === | === Phase 1: Kickoff and Test Plan sign off === | ||
[[File:qa_phase1.png|Phase 1]] | |||
* <u>QA requests a feature walkthrough from the engineering team responsible with the feature</u> | * <u>QA requests a feature walkthrough from the engineering team responsible with the feature</u> | ||
** '''Description''': the walkthrough can consist of an actual demo held by the engineering team, a discussion based on a very specific agenda or an email thread tackling a very specific list of topics/questions essential for sketching a Test Plan | ** '''Description''': the walkthrough can consist of an actual demo held by the engineering team, a discussion based on a very specific agenda or an email thread tackling a very specific list of topics/questions essential for sketching a Test Plan | ||
| Line 27: | Line 27: | ||
** '''References''': https://goo.gl/HdgWft (Test Plan template) | ** '''References''': https://goo.gl/HdgWft (Test Plan template) | ||
* <u>QA sends the draft Test Plan to | * <u>QA sends the draft Test Plan to Engineering, Product and QA Manager for review and formal sign off</u> | ||
** '''When''': as soon as the draft Test Plan has been green-lit internally, by QA peers and/or Team Lead | ** '''When''': as soon as the draft Test Plan has been green-lit internally, by QA peers and/or Team Lead | ||
** '''References''': https://goo.gl/NyG3RF (Email template) | |||
=== Phase 2: Test preparation and scheduling === | === Phase 2: Test preparation and scheduling === | ||
* <u>If the draft Test Plan has been signed off by the | [[File:qa_phase2.png|Phase 2]] | ||
** '''When''': immediately after the Test Plan has been formally signed off | * <u>If the draft Test Plan has been signed off by the required parties, QA starts creating high-level test cases based on the Test Objectives described in the Test Plan</u> | ||
** '''When''': immediately after the Test Plan has been formally signed off by required parties | |||
** '''Exceptions''': Test Suite drafting could also start while the Test Plan is being reviewed, if QA is confident about the Test Objectives proposed in the Test Plan | ** '''Exceptions''': Test Suite drafting could also start while the Test Plan is being reviewed, if QA is confident about the Test Objectives proposed in the Test Plan | ||
| Line 46: | Line 48: | ||
=== Phase 3: Mid Nightly test execution and sign off === | === Phase 3: Mid Nightly test execution and sign off === | ||
[[File:qa_phase3.png|Phase 3]] | |||
* <u>QA sends out a preliminary status report and schedule update before the mid-Nightly feature sign off</u> | * <u>QA sends out a preliminary status report and schedule update before the mid-Nightly feature sign off</u> | ||
** '''Goal''': For QA to discuss the status of the feature with the Engineering team, prior to the formal sign off. | ** '''Goal''': For QA to discuss the status of the feature with the Engineering team, prior to the formal sign off. | ||
| Line 54: | Line 57: | ||
** '''When''': by end of week #3 (6-week or 7-week release cycle) or by end of week #4 (8-week cycle) | ** '''When''': by end of week #3 (6-week or 7-week release cycle) or by end of week #4 (8-week cycle) | ||
** '''Best practice''': while not mandatory, it’s usually best to consult with the Team Lead if the status or arguments used in the feature sign off report are unclear or might be subject to interpretation | ** '''Best practice''': while not mandatory, it’s usually best to consult with the Team Lead if the status or arguments used in the feature sign off report are unclear or might be subject to interpretation | ||
** '''References''': https://goo.gl/ | ** '''References''': https://goo.gl/AEY99U (sign off template) | ||
=== Phase 4: Pre Beta test execution and sign off === | === Phase 4: Pre Beta test execution and sign off === | ||
[[File:qa_phase4.png|Phase 4]] | |||
* <u>QA sends out a preliminary status report and schedule update 1 week before the pre-Beta sign-off</u> | * <u>QA sends out a preliminary status report and schedule update 1 week before the pre-Beta sign-off</u> | ||
** '''Goal''': For QA to discuss the status of the feature with the Engineering team, prior to the formal sign off. | ** '''Goal''': For QA to discuss the status of the feature with the Engineering team, prior to the formal sign off. | ||
| Line 63: | Line 67: | ||
* <u>QA formally signs off the feature pre-Beta</u> | * <u>QA formally signs off the feature pre-Beta</u> | ||
** '''When''': by end of week #5 (6-week), week #6 (7-week cycle) or by end of week #7 (8-week cycle) -- or 1 week before merge day | ** '''When''': by end of week #5 (6-week), week #6 (7-week cycle) or by end of week #7 (8-week cycle) -- or 1 week before merge day | ||
** '''References''': https://goo.gl/ | ** '''References''': https://goo.gl/AEY99U (sign off template) | ||
=== Phase 5: Pre Release test execution and sign off === | === Phase 5: Pre Release test execution and sign off === | ||
[[File:qa_phase5.png|Phase 5]] | |||
* <u>(not always applicable) Engineering QA hands over the feature to Release QA and informs the engineering team about it</u> | * <u>(not always applicable) Engineering QA hands over the feature to Release QA and informs the engineering team about it</u> | ||
** '''When''': in the 1st week of the release cycle | ** '''When''': in the 1st week of the release cycle | ||
| Line 77: | Line 82: | ||
* <u>QA formally signs off the feature pre-Release</u> | * <u>QA formally signs off the feature pre-Release</u> | ||
** '''When''': by end of week #4 (6-week cycle), by end of week #5 (7-week cycle) or by end of week #6 (8-week cycle) -- or 2 weeks before merge day | ** '''When''': by end of week #4 (6-week cycle), by end of week #5 (7-week cycle) or by end of week #6 (8-week cycle) -- or 2 weeks before merge day | ||
** '''References''': https://goo.gl/ | ** '''References''': https://goo.gl/AEY99U (sign off template) | ||
=== Time table === | === Time table === | ||
| Line 83: | Line 88: | ||
{| class="wikitable" | {| class="wikitable" | ||
|- | |- | ||
! WEEK NO. !! WEEK #1 <br/>(1st new Beta) !! WEEK #2 !! WEEK #3 !! WEEK #4 !! WEEK #5 !! WEEK #6 <br/>(1st RC) | ! WEEK NO. !! width=100px|WEEK #1 <br/>(1st new Beta) !! width=100px|WEEK #2 !! width=100px|WEEK #3 !! width=100px|WEEK #4 !! width=100px|WEEK #5 !! width=100px|WEEK #6 <br/>(1st RC) | ||
|- | |- | ||
| '''NIGHTLY''' || || Preliminary feature status report || Mid Nightly feature sign offs || Preliminary feature status report || Pre Beta feature sign offs || | | '''NIGHTLY''' || || Preliminary feature status report || Mid Nightly feature sign offs || Preliminary feature status report || Pre Beta feature sign offs || | ||
| Line 93: | Line 98: | ||
{| class="wikitable" | {| class="wikitable" | ||
|- | |- | ||
! WEEK NO. !! WEEK #1 <br/>(1st new Beta) !! WEEK #2 !! WEEK #3 !! WEEK #4 !! WEEK #5 !! WEEK #6 !! WEEK #7 <br/>(1st RC) | ! WEEK NO. !! width=100px|WEEK #1 <br/>(1st new Beta) !! width=100px|WEEK #2 !! width=100px|WEEK #3 !! width=100px|WEEK #4 !! width=100px|WEEK #5 !! width=100px|WEEK #6 !! width=100px|WEEK #7 <br/>(1st RC) | ||
|- | |- | ||
| '''NIGHTLY''' || || Preliminary feature status report || Mid Nightly feature sign offs || || Preliminary feature status report || Pre Beta feature sign offs || | | '''NIGHTLY''' || || Preliminary feature status report || Mid Nightly feature sign offs || || Preliminary feature status report || Pre Beta feature sign offs || | ||
| Line 104: | Line 109: | ||
{| class="wikitable" | {| class="wikitable" | ||
|- | |- | ||
! WEEK NO. !! WEEK #1 <br/>(1st new Beta) !! WEEK #2 !! WEEK #3 !! WEEK #4 !! WEEK #5 !! WEEK #6 !! WEEK #7 !! WEEK #8 <br/>(1st RC) | ! WEEK NO. !! width=100px|WEEK #1 <br/>(1st new Beta) !! width=100px|WEEK #2 !! width=100px|WEEK #3 !! width=100px|WEEK #4 !! width=100px|WEEK #5 !! width=100px|WEEK #6 !! width=100px|WEEK #7 !! width=100px|WEEK #8 <br/>(1st RC) | ||
|- | |- | ||
| '''NIGHTLY''' || || || Preliminary feature status report || Mid Nightly feature sign offs || || Preliminary feature status report || Pre Beta feature sign offs || | | '''NIGHTLY''' || || || Preliminary feature status report || Mid Nightly feature sign offs || || Preliminary feature status report || Pre Beta feature sign offs || | ||
| Line 122: | Line 127: | ||
== Sign off overview == | == Sign off overview == | ||
{{warning| QA’s sign off is based on the quality status of the feature in question. The green, yellow or red quality status is a schedule-based conversation, ''not'' a general release-readiness status.}} | {{warning| QA’s sign off is based on the quality status of the feature in question. The green, yellow or red quality status is a schedule-based conversation, ''not'' a general release-readiness status.}} | ||
=== Quality status evaluation criteria === | === Quality status evaluation criteria === | ||
| Line 174: | Line 178: | ||
<br/> | <br/> | ||
<u>Quality trend</u>: | <u>Quality trend</u>: | ||
* mid-Nightly: RED (signed off on Aug 30, 2017 - [https://testrail.stage.mozaws.net/index.php?/plans/view/5795 test report]) | * mid-Nightly: <font style="background: #FF0000;">RED</font> (signed off on Aug 30, 2017 - [https://testrail.stage.mozaws.net/index.php?/plans/view/5795 test report]) | ||
* pre-Beta: YELLOW (signed off on Sep 16, 2017 - [https://testrail.stage.mozaws.net/index.php?/plans/view/6196 test report]) | * pre-Beta: <font style="background: #FFFF00;>YELLOW</font> (signed off on Sep 16, 2017 - [https://testrail.stage.mozaws.net/index.php?/plans/view/6196 test report]) | ||
<br/> | <br/> | ||
<u>Testing summary</u>: | <u>Testing summary</u>: | ||
| Line 183: | Line 187: | ||
* Firefox 57 feature QA status: https://goo.gl/i6q3VA | * Firefox 57 feature QA status: https://goo.gl/i6q3VA | ||
<br/> | <br/> | ||
'''What’s this report about?''' Find out [https://wiki.mozilla.org/QA/Feature_Testing here]. | '''What’s this report about?''' Find out [https://wiki.mozilla.org/QA/Feature_Testing here]. | ||
| Line 213: | Line 216: | ||
<br/><br/> | <br/><br/> | ||
<u>Quality status</u>: <font style="background: #FF0000;>RED</font> | <u>Quality status</u>: <font style="background: #FF0000;>RED</font> | ||
* '''Why is this feature | * '''Why is this feature red?''' | ||
** Credit Card information sync has not been implemented yet (see [https://bugzil.la/1395028 Bug 1395028]). | ** Credit Card information sync has not been implemented yet (see [https://bugzil.la/1395028 Bug 1395028]). | ||
** Form Autofill does not work for some of the top shopping sites (see [https://bugzil.la/1398101 Bug 1398101]), such as Groupon and Newegg. | ** Form Autofill does not work for some of the top shopping sites (see [https://bugzil.la/1398101 Bug 1398101]), such as Groupon and Newegg. | ||
| Line 229: | Line 232: | ||
* Firefox 57 feature QA status: https://goo.gl/i6q3VA | * Firefox 57 feature QA status: https://goo.gl/i6q3VA | ||
<br/> | <br/> | ||
'''What’s this report about?''' Find out [https://wiki.mozilla.org/QA/Feature_Testing here]. | '''What’s this report about?''' Find out [https://wiki.mozilla.org/QA/Feature_Testing here]. | ||
| Line 237: | Line 239: | ||
</blockquote> | </blockquote> | ||
=== Quality | ==== Pref'd OFF ==== | ||
===== Description ===== | |||
Features riding a specific Firefox version in a disabled state (via preference flip) will not be receiving a sign off report from QA. This is because there’s no real impact for the end user, and so an in-depth assessment of that feature’s quality is irrelevant for the Firefox version in question.<br/><br/> | |||
Instead of a sign off, QA will send out a regular test report describing what was considered in scope/covered. | |||
===== Example ===== | |||
This is an email report example of a regular test report. | |||
* <u>Subject</u>: | |||
<blockquote style="background: #F3F3F3; border-left: 4px solid #ccc; padding-left: 15px;">[desktop] [mobile] [feature] Structured Sync Bookmark Application - Test Results</blockquote> | |||
* <u>Body</u>: | |||
<blockquote style="background: #F3F3F3; border-left: 4px solid #ccc; padding-left: 15px;">Hello,<br/> | |||
I have completed the testing for Structured Bookmarks Application feature. Please see below the details:<br/> | |||
The testing was done on following platforms: | |||
* Mac OS 10.13 | |||
* Windows 10 | |||
* Ubuntu 16.0.4 | |||
* iOS | |||
* Android | |||
Testing summary: | |||
* Test Plan: https://docs.google.com/document/d/15JYYcDCz_wDohD-6gcwnNhqn_seWfHP_CfXfdKcH2eA/edit?usp=sharing | |||
* Test report: https://testrail.stage.mozaws.net/index.php?/reports/view/824 | |||
* New bugs: https://bugzilla.mozilla.org/buglist.cgi?bug_id=1436888%2C%201437153%2C%201435446%2C%201442805%2C%201442467%2C1440518&bug_id_type=anyexact&resolution=---&list_id=14376107 | |||
<br/> | |||
'''What’s this report about?''' Find out [https://wiki.mozilla.org/QA/Feature_Testing here]. | |||
<br/><br/> | |||
Please note that these are just the testing results for the verification that we completed recently on Nightly 60. It is not a formal sign-off as the feature will be pref'd off by default in beta 60. Please feel free to contact me in case of any questions/clarifications.<br/> | |||
Thank you,<br/> | |||
Kanchan Kumari<br/> | |||
Engineering QA | |||
</blockquote> | |||
==== Bugwork ==== | |||
===== Description ===== | |||
Minor changes (or low impact) to Firefox, mainly consisting of a meta-bug and a few dependencies are usually decided together with the Eng Teams to be treated as bugworks instead of features. This also applies to low impact features coming as a testing request late in the cycle, which don't allow the QA teams to apply the full testing process usually needed for a feature (Test Plan creation, sign-off emails..). | |||
Bugworks will not be receiving a standard sign-off report from QA, but only a test report (without a Quality Status - sign-off color, Quality Trend..) describing what was considered in scope/covered. | |||
<br/><br/> | |||
===== Example ===== | |||
This is an email report example of a regular bugwork test report. | |||
* <u>Subject</u>: | |||
<blockquote style="background: #F3F3F3; border-left: 4px solid #ccc; padding-left: 15px;">[62][desktop][bugwork] Widevine CDM 1.4.9.1088 - Beta 62 test report (2018-08-16)</blockquote> | |||
* <u>Body</u>: | |||
<blockquote style="background: #F3F3F3; border-left: 4px solid #ccc; padding-left: 15px;">Hello,<br/> | |||
We have completed the testing for Widevine CDM 1.4.9.1088 feature. Please see below the details. | |||
Testing covered the following: | |||
* Firefox builds: Fx 62.0b16 | |||
* Platforms: Windows 7 x86, Windows 10 x64, Ubuntu 16.04 x64, Mac OS X 10.13, Windows 8 x86 | |||
Testing summary: | |||
* Test Plan: N/A | |||
* Test report: https://testrail.stage.mozaws.net/index.php?/reports/view/1073 | |||
* Bugs: https://bugzilla.mozilla.org/buglist.cgi?quicksearch=1482918%2C%20653826%2C%201482161&list_id=14433425 | |||
Please note that this email stands as a test report for “Widevine CDM 1.4.9.1088” feature on Beta 62. Testing was performed as requested on different sites to ensure their functionality and stability while Widevine CDM 1.4.9.1088 is enabled. The feature functionality and stability are looking good, since we did not encounter any crashes and the functionality issues we found are not major. | |||
Thanks, | |||
</blockquote> | |||
=== Sign off template === | |||
The feature sign off template is available [https://docs.google.com/document/d/11sCQTPO2F2xWObcBkbXh4VPp9ampujujVvO7XJh7sqA/edit# here]. | The feature sign off template is available [https://docs.google.com/document/d/11sCQTPO2F2xWObcBkbXh4VPp9ampujujVvO7XJh7sqA/edit# here]. | ||
== Useful references == | |||
* All QA documentation available for features, structured per Firefox version: [https://drive.google.com/open?id=0B-n79IWvmJYYQngxVjhiRUJ1bTA here]. | |||
* Feature QA status per Firefox version: | |||
** Firefox 66: [https://docs.google.com/spreadsheets/d/1vPyFxPmnQt5YZThJlFaeyriLkeXU6sE6jgEHgAryUbo/edit#gid=0 here] | |||
** Firefox 65: [https://docs.google.com/spreadsheets/d/1-zxhm8GJf9fZ0kFbrfB5tzpfCcTqUNHVhDJaCfIr89o/edit here] | |||
** Firefox 64: [https://docs.google.com/spreadsheets/d/1z4_GjF-OHKihIeqfJGzH3NUC0hK3Ct8chyHOm8Nguo4/edit here] | |||
** Firefox 63: [https://docs.google.com/spreadsheets/u/2/d/1IdIMWOrYymTVKgCPiPoxWQ3xlYBNbGX_UC2rAX1YRYY/edit here] | |||
** Firefox 62: [https://docs.google.com/document/u/2/d/1rxisBtzIVZwSAuIIzQ5WCAUUYL2ZhtUQ9xiSBBAOCKY/edit here] | |||
** Firefox 61: [https://docs.google.com/document/d/1IjualMcpheEbZsT7ujyEr0tg4XIpdaG4oRpu1MTUQw0/edit here] | |||
** Firefox 60: [https://docs.google.com/document/d/1jLrir7BEF7nerypyqbbtdu_uL2IDRLjZqigSOeUMFho/edit here] | |||
** Firefox 59: [https://docs.google.com/document/d/1HLCYgwnFFj3sIBybrFM19VhccSaLuqwXaRPaGHj1Ap0/edit here] | |||
** Firefox 58: [https://docs.google.com/document/d/1WIkZv89twzb0oLWsyO78FkWxPMd3qJwKXxP00wM1c-g/edit here] | |||
** Firefox 57: [https://docs.google.com/document/u/2/d/1e840tyKTmhBdlYxtpWsGnyVXgyKVSJxxv9p98mMDqEU/edit here] | |||