Releases/Firefox 68/Test Plan: Difference between revisions

Releases/Firefox 68/Test Plan (view source)

Revision as of 07:43, 5 April 2019

8,198 bytes added , 5 April 2019

m

-

Andrei.vaida

Confirmed users

1,595

edits

@@ Line 17: / Line 17: @@
 === Communication channels ===
-* Slack: [https://mozilla.slack.com/messages/CAC9YSH3P #qa-coordination]
+* '''Slack''': [https://mozilla.slack.com/messages/CAC9YSH3P #qa-coordination]
 === Firefox for Desktop ===
-<div class="toccolours mw-collapsible" style="width:auto; overflow:auto;">
+<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
 <div style="font-weight:bold;line-height:1.6;">Engineering QA</div>
 <div class="mw-collapsible-content">
@@ Line 30: / Line 30: @@
 </div></div>
-<div class="toccolours mw-collapsible" style="width:auto; overflow:auto;">
+<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
 <div style="font-weight:bold;line-height:1.6;">Release QA</div>
 <div class="mw-collapsible-content">
@@ Line 36: / Line 36: @@
 * '''Softvision Manager''': [https://phonebook.mozilla.org/tree.php?search/avaida@mozilla.com Andrei Vaida]
 * '''Team Leads''': [https://phonebook.mozilla.org/tree.php?search/bmaris@mozilla.com Bogdan Maris], [https://phonebook.mozilla.org/tree.php?search/cionce@mozilla.com Cornel Ionce]
-* '''Office''': Romania (GMT+2)
+* '''Office''': Romania (EEST)
 * '''Responsible for''': new feature testing, Beta build readiness, Developer Edition build readiness, ESR build readiness, RC build readiness, Dot release build readiness, Chemspill build readiness, bug verification, bug investigation
+</div></div>
+=== Firefox for Mobile ===
+<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
+<div style="font-weight:bold;line-height:1.6;">Release QA</div>
+<div class="mw-collapsible-content">
+* '''Reporting to''': [https://phonebook.mozilla.org/tree.php?search/njpark@mozilla.com No-Jun Park]
+* '''Softvision Manager''': [https://phonebook.mozilla.org/tree.php?search/ichiorean@mozilla.com Ioana Chiorean]
+* '''Team Lead''': [https://phonebook.mozilla.org/tree.php?search/sflorean@mozilla.com Sorina Florean]
+* '''Office''': Romania (EEST)
+* '''Responsible for''': <mark>todo</mark>
+</div></div>
+<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
+<div style="font-weight:bold;line-height:1.6;">Firefox for iOS QA</div>
+<div class="mw-collapsible-content">
+* '''Reporting to''': [https://phonebook.mozilla.org/tree.php?search/njpark@mozilla.com No-Jun Park]
+* '''Softvision Manager''': [https://phonebook.mozilla.org/tree.php?search/ichiorean@mozilla.com Ioana Chiorean]
+* '''Team Lead''': [https://phonebook.mozilla.org/tree.php?search/csuciu@mozilla.com Catalin Suciu]
+* '''Office''': Romania (EEST)
+* '''Responsible for''': <mark>todo</mark>
+</div></div>
+=== AMO, WebExtensions ===
+<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
+<div style="font-weight:bold;line-height:1.6;">Add-ons QA</div>
+<div class="mw-collapsible-content">
+* '''Reporting to''': [https://phonebook.mozilla.org/tree.php?search/kraj@mozilla.com Krupa Raj]
+* '''Softvision Manager''': [https://phonebook.mozilla.org/tree.php?search/rbologa@mozilla.com Rares Bologa]
+* '''Team Leads''': [https://phonebook.mozilla.org/tree.php?search/vcarciu@mozilla.com Victor Carciu], [https://phonebook.mozilla.org/tree.php?search/mcotetiu@mozilla.com Madalin Cotetiu]
+* '''Office''': Romania (EEST)
+* '''Responsible for''': <mark>todo</mark>
+</div></div>
+=== SHIELD, Normandy, Subscription services ===
+<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
+<div style="font-weight:bold;line-height:1.6;">Experiments QA</div>
+<div class="mw-collapsible-content">
+* '''Reporting to''': [https://phonebook.mozilla.org/tree.php?search/kraj@mozilla.com Krupa Raj]
+* '''Softvision Manager''': [https://phonebook.mozilla.org/tree.php?search/rbologa@mozilla.com Rares Bologa]
+* '''Team Leads''': [https://phonebook.mozilla.org/tree.php?search/poiegas@mozilla.com Paul Oiegas]
+* '''Office''': Romania (EEST), Argentina (GMT-3)
+* '''Responsible for''': <mark>todo</mark>
+</div></div>
+== QA activities ==
+=== New feature readiness ===
+All new features tested by QA and shipping with Firefox 68 are available [https://docs.google.com/spreadsheets/d/1ygCmrDTQmoN6ecyNlkPHft6O2tayhn3S1RN5ii_ER_8/edit#gid=0 here].
+The feature testing process followed by QA is available [https://wiki.mozilla.org/QA/Feature_Testing_v2 here].
+=== Bug fix verification ===
+* '''Daily verification of uplifted bugs''' - these are uplifts, triaged daily and verified under ~48hrs
+* '''Ongoing verification of Nightly bugs''' - this is the general bug verification done throughout the Nightly cycle; it sometimes overlaps with uplifts verification; also involves active monitoring of <code>qe-verify?</code> bugs
+* '''Ongoing verification of Beta bugs''' - this is the general bug verification done throughout the Beta cycle; it sometimes overlaps with uplifts verification; also involves active monitoring of <code>qe-verify?</code> bugs
+* '''On-demand verification of bugs''' - these are special requests (ideally coming in through PI, but also through Bugzilla needinfo, chat or emails)
+* '''Periodical verification of security bugs''' - these are ''post-critsmash'' bugs, triaged and verified ~every 2wks
+=== Unconfirmed bug triage ===
+This is the triage conducted by Engineering QA.
+=== Security bug triage ===
+This is the post-CritSmash triage conducted by Engineering QA (on Nightly) and Release QA (on Beta, Release, ESR).
+=== Bug investigation ===
+This is the daily check and investigation of new <code>regressionwindow-wanted</code> bugs, conducted by Engineering QA and Release QA.
+== Nightly 68 milestones ==
+=== [early/mid cycle] Nightly build readiness ===
+'''Scope''': assess build readiness for shipping to Beta by testing Nightly periodically and providing a list of issues (if any) that need to be addressed before the merge ([https://docs.google.com/document/d/1etTi-uvpAlaoAoErcHcJVvd9-QIcdn7uH-WCzu72T3Y/edit more on this]).
+<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
+<div style="font-weight:bold;line-height:1.6;">Nightly 68.0a1 <date1></div>
+<div class="mw-collapsible-content">
+* '''Build''': 68.0a1 (commit, ID) - Sign off_Status
+** {{mnotstarted|}} Functional testing (manual) → report
+*** New bugs – regressions: ID1, ID2, IDN  |  non-regressions: ID1, ID2, IDN
+*** Known bugs – high-impact: ID1, ID2, IDN  |  medium/low-impact: <bug list>
+</div></div>
+=== [prior to merge] Nightly regression testing ===
+<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
+<div style="font-weight:bold;line-height:1.6;">Nightly 68.0a1 <date1></div>
+<div class="mw-collapsible-content">
+* '''Build''': 68.0a1 (commit, ID) - Sign off_Status
+** {{mnotstarted|}} Regression testing (manual) → report
+*** New bugs – regressions: ID1, ID2, IDN  |  non-regressions: ID1, ID2, IDN
+*** Known bugs – high-impact: ID1, ID2, IDN  |  medium/low-impact: <bug list>
+</div></div>
+=== [optional] [prior to merge] Nightly new feature smoke testing ===
+'''Note''': this is optional, based on available QA bandwidth.
+== Beta/DevEd 68 milestones ==
+=== Developer Edition build readiness ===
+'''Scope''': assess build stability periodically (around every 2 weeks) to make sure we are shipping quality products to our Developer Edition users from the aurora update channel ([https://docs.google.com/document/d/1BsZ7QTzomPKBgRRYLnTcuAcrJM7BOcK2SgMtIoVlWNY/edit# more on this]).
+<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
+<div style="font-weight:bold;line-height:1.6;">DevEd 68.0b1</div>
+<div class="mw-collapsible-content">
+* '''Build''': 68.0b1-build1 (commit, ID) - Sign off_Status
+** '''Functional tests (manual)''':
+*** {{mnotstarted|}} Smoke, Regression and Exploratory testing on _ → report
+*** {{mnotstarted|}} Bug fix verification – <bug list>
+** '''Update tests (manual)''':
+*** {{mnotstarted|}} aurora-cdntest → report
+*** {{mnotstarted|}} aurora → report
+</div></div>
+=== Beta build readiness ===
+<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
+<div style="font-weight:bold;line-height:1.6;">Beta 68.0b3</div>
+<div class="mw-collapsible-content">
+* Build: 68.0b3-build1 (commit, ID) - Sign off_Status
+** '''Functional tests (manual)''':
+*** {{mnotstarted|}} Smoke, Regression and Exploratory testing on _ → report
+*** {{mnotstarted|}} Smoke testing on Ubuntu snap package → report
+*** {{mnotstarted|}} Smoke testing on DLL interceptor → report
+*** {{mnotstarted|}} Bug fix verification – <bug list>
+** '''Update tests (manual)''':
+*** {{mnotstarted|}} beta-cdntest, aurora-cdntest → report
+*** {{mnotstarted|}} beta, aurora → report
+</div></div>
+== Release 68 milestones ==
+<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
+<div style="font-weight:bold;line-height:1.6;">RC1 68.0</div>
+<div class="mw-collapsible-content">
+* '''Build''': 68.0-build1 (commit, ID) - Sign off_Status
+** '''Functional tests (manual)''':
+*** {{mnotstarted|}} Smoke, Regression and Exploratory testing on _ → report
+*** {{mnotstarted|}} Smoke testing on Ubuntu snap package → report
+*** {{mnotstarted|}} Smoke testing on Windows Acer builds → report
+*** {{mnotstarted|}} Bug fix verification – <bug list>
+** '''Update tests (manual)''':
+*** {{mnotstarted|}} beta-cdntest → report
+*** {{mnotstarted|}} beta → report
+*** {{mnotstarted|}} release-localtest → report
+*** {{mnotstarted|}} release-cdntest → report
+*** {{mnotstarted|}} release → report
 </div></div>