MozillaWiki

Releases/Firefox 68/Test Plan: Difference between revisions

< Releases‎ | Firefox 68
(-)
 
 
(118 intermediate revisions by 19 users not shown)
Line 17: Line 17:


=== Communication channels ===
=== Communication channels ===
* Slack: [https://mozilla.slack.com/messages/CAC9YSH3P #qa-coordination]
* '''Slack''': [https://mozilla.slack.com/messages/CAC9YSH3P #qa-coordination]


=== Firefox for Desktop ===
=== Firefox for Desktop ===
<div class="toccolours mw-collapsible" style="width:auto; overflow:auto;">
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">Engineering QA</div>
<div style="font-weight:bold;line-height:1.6;">Engineering QA</div>
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
Line 30: Line 30:
</div></div>
</div></div>


<div class="toccolours mw-collapsible" style="width:auto; overflow:auto;">
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">Release QA</div>
<div style="font-weight:bold;line-height:1.6;">Release QA</div>
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
Line 36: Line 36:
* '''Softvision Manager''': [https://phonebook.mozilla.org/tree.php?search/avaida@mozilla.com Andrei Vaida]
* '''Softvision Manager''': [https://phonebook.mozilla.org/tree.php?search/avaida@mozilla.com Andrei Vaida]
* '''Team Leads''': [https://phonebook.mozilla.org/tree.php?search/bmaris@mozilla.com Bogdan Maris], [https://phonebook.mozilla.org/tree.php?search/cionce@mozilla.com Cornel Ionce]
* '''Team Leads''': [https://phonebook.mozilla.org/tree.php?search/bmaris@mozilla.com Bogdan Maris], [https://phonebook.mozilla.org/tree.php?search/cionce@mozilla.com Cornel Ionce]
* '''Office''': Romania (GMT+2)
* '''Office''': Romania (EEST)
* '''Responsible for''': new feature testing, Beta build readiness, Developer Edition build readiness, ESR build readiness, RC build readiness, Dot release build readiness, Chemspill build readiness, bug verification, bug investigation
* '''Responsible for''': new feature testing, Beta build readiness, Developer Edition build readiness, ESR build readiness, RC build readiness, Dot release build readiness, Chemspill build readiness, bug verification, bug investigation
</div></div>
=== Firefox for Mobile ===
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">Release QA</div>
<div class="mw-collapsible-content">
* '''Reporting to''': [https://phonebook.mozilla.org/tree.php?search/njpark@mozilla.com No-Jun Park]
* '''Softvision Manager''': [https://phonebook.mozilla.org/tree.php?search/ichiorean@mozilla.com Ioana Chiorean]
* '''Team Lead''': [https://phonebook.mozilla.org/tree.php?search/sflorean@mozilla.com Sorina Florean]
* '''Office''': Romania (EEST)
* '''Responsible for''': <mark>todo</mark>
</div></div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">Firefox for iOS QA</div>
<div class="mw-collapsible-content">
* '''Reporting to''': [https://phonebook.mozilla.org/tree.php?search/njpark@mozilla.com No-Jun Park]
* '''Softvision Manager''': [https://phonebook.mozilla.org/tree.php?search/ichiorean@mozilla.com Ioana Chiorean]
* '''Team Lead''': [https://phonebook.mozilla.org/tree.php?search/csuciu@mozilla.com Catalin Suciu]
* '''Office''': Romania (EEST)
* '''Responsible for''': <mark>todo</mark>
</div></div>
=== AMO, WebExtensions ===
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">Add-ons QA</div>
<div class="mw-collapsible-content">
* '''Reporting to''': [https://phonebook.mozilla.org/tree.php?search/kraj@mozilla.com Krupa Raj]
* '''Softvision Manager''': [https://phonebook.mozilla.org/tree.php?search/rbologa@mozilla.com Rares Bologa]
* '''Team Leads''': [https://phonebook.mozilla.org/tree.php?search/vcarciu@mozilla.com Victor Carciu], [https://phonebook.mozilla.org/tree.php?search/mcotetiu@mozilla.com Madalin Cotetiu]
* '''Office''': Romania (EEST)
* '''Responsible for''': AMO feature testing, AMO weekly release, AMO Sanity, AMO bug verification, Webextensions features, Webextensions bug triage and verification, Telemetry
</div></div>
=== Experiments, Normandy, Subscription services, and other Firefox features ===
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">Experiments QA</div>
<div class="mw-collapsible-content">
* '''Reporting to''': [https://phonebook.mozilla.org/tree.php?search/kraj@mozilla.com Krupa Raj]
* '''Softvision Manager''': [https://phonebook.mozilla.org/tree.php?search/rbologa@mozilla.com Rares Bologa]
* '''Team Leads''': [https://phonebook.mozilla.org/tree.php?search/poiegas@mozilla.com Paul Oiegas]
* '''Office''': Romania (EEST)
* '''Responsible for''': testing Experiments, Normandy, Experimenter website, Activity Stream, Screenshots, Send (Web & Android) and Lockbox Desktop.
</div></div>
== QA activities ==
=== New feature readiness ===
All new features tested by QA and shipping with Firefox 68 are available [https://docs.google.com/spreadsheets/d/1ygCmrDTQmoN6ecyNlkPHft6O2tayhn3S1RN5ii_ER_8/edit#gid=0 here].
The feature testing process followed by QA is available [https://wiki.mozilla.org/QA/Feature_Testing_v2 here].
=== Bug fix verification ===
* '''Daily verification of uplifted bugs''' - these are uplifts, triaged daily and verified under ~48hrs
* '''Ongoing verification of Nightly bugs''' - this is the general bug verification done throughout the Nightly cycle; it sometimes overlaps with uplifts verification; also involves active monitoring of <code>qe-verify?</code> bugs
* '''Ongoing verification of Beta bugs''' - this is the general bug verification done throughout the Beta cycle; it sometimes overlaps with uplifts verification; also involves active monitoring of <code>qe-verify?</code> bugs
* '''On-demand verification of bugs''' - these are special requests (ideally coming in through PI, but also through Bugzilla <code>needinfo?</code>, chat or emails)
* '''Periodical verification of security bugs''' - these are ''post-critsmash'' bugs, triaged and verified ~every 2wks
=== Unconfirmed bug triage ===
This is the triage conducted by Engineering QA.
=== Security bug triage ===
This is the post-CritSmash triage conducted by Engineering QA (on Nightly) and Release QA (on Beta, Release, ESR).
=== Bug investigation ===
This is the daily check and investigation of new <code>regressionwindow-wanted</code> bugs, conducted by Engineering QA and Release QA.
== Nightly 68 milestones ==
=== [early/mid cycle] Nightly build readiness ===
'''Scope''': assess build readiness for shipping to Beta by testing Nightly periodically and providing a list of issues (if any) that need to be addressed before the merge ([https://docs.google.com/document/d/1etTi-uvpAlaoAoErcHcJVvd9-QIcdn7uH-WCzu72T3Y/edit more on this]).
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">Nightly 68.0a1 (2019-04-04)</div>
<div class="mw-collapsible-content">
* '''Build''': 68.0a1 (commit, ID) - <font style="color:green; font-weight:bold;">GREEN</font>
** {{mdone|}} Functional testing (manual) → [https://testrail.stage.mozaws.net/index.php?/plans/view/16518 report]
*** New bugs
**** Regressions: 1541400
**** Non-regressions: 1541376, 1541345, 1541057
*** Known bugs
**** High-impact: none
**** Low/medium-impact: [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=1260585%2C1144124%2C1037792%2C911804%2C1532730%2C1536756%2C1195242%2C1393460%2C1525585%2C1476585%2C1515289%2C1515332%2C1465445%2C1529216%2C1537769%2C1511869%2C1511999%2C1396732%2C1441881%2C1443511%2C1194214%2C1481496%2C1517548%2C1354992&list_id=14649953 bug list]
</div></div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">Nightly 68.0a1 (2019-04-18)</div>
<div class="mw-collapsible-content">
* '''Build''': 68.0a1 (commit, ID) - <font style="color:green; font-weight:bold;">GREEN</font>
** {{mdone|}} Functional testing (manual) → [https://testrail.stage.mozaws.net/index.php?/plans/view/16879 report]
*** New bugs
**** Regressions: 1544509
**** Non-regressions: 1544701
*** Known bugs
**** High-impact: none
**** Low/medium-impact: [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=1260585%2C1144124%2C1037792%2C911804%2C1532730%2C1536756%2C1195242%2C1393460%2C1525585%2C1476585%2C1515289%2C1515332%2C1465445%2C1529216%2C1537769%2C1511869%2C1511999%2C1396732%2C1441881%2C1443511%2C1194214%2C1481496%2C1517548%2C1354992&list_id=14649953 bug list]
</div></div>
=== [prior to merge] Nightly regression testing ===
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">Nightly 68.0a1 (2019-05-16)</div>
<div class="mw-collapsible-content">
* '''Build''': 68.0a1 (commit, ID) - <font style="color:green; font-weight:bold;">GREEN</font>
** {{mdone|}} Regression testing (manual) → [https://testrail.stage.mozaws.net/index.php?/plans/view/17266 report]
*** New bugs
**** Regressions: 1550523, 1551554, 1551215, 1551452
**** Non-regressions: 1544509, 1549360, 1550459, 1550533, 1551534
*** Known bugs
**** High-impact: none
**** Low/medium-impact: [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=1354992%2C%201370743%2C%201511869%2C%201481496%2C%201445644%2C%201360545%2C%201517548%2C%201511999%2C%201378105%2C%201470409%2C%201430114%2C%201358102%2C%201487467%2C%201328984%2C%201288773%2C%201406899%2C%201194214%2C%201441881%2C%201308007%2C%201515289%2C%201396732%2C%201485135%2C%201443511%2C%201470152%2C%201491251%2C%201527607%2C1393460%2C%201532730%2C%201512611%2C%201282075%2C%201528958%2C%201525607%2C%201525585%2C%20558178%2C%201413233%2C%201527257%2C%201509317%2C%201423615%2C%201198363%2C%201533409%2C%201496429%2C%201547806%2C%20169132%2C%201443145%2C%201428269%2C%201496472%2C%201428354&list_id=14766802 bug list]
</div></div>
=== [optional] [prior to merge] Nightly new feature smoke testing ===
'''Note''': this is optional, based on available QA bandwidth.
== Beta/DevEd 68 milestones ==
=== Developer Edition build readiness ===
'''Scope''': assess build stability periodically (around every 2 weeks) to make sure we are shipping quality products to our Developer Edition users from the aurora update channel ([https://docs.google.com/document/d/1BsZ7QTzomPKBgRRYLnTcuAcrJM7BOcK2SgMtIoVlWNY/edit# more on this]).
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">DevEd 68.0b1</div>
<div class="mw-collapsible-content">
* '''Build''': 68.0b1-build1 ([https://hg.mozilla.org/releases/mozilla-beta/rev/d313e24f18c2 d313e24f18c2], 20190513111358) - <font style="color:green; font-weight:bold;">GREEN</font>
** '''Functional tests (manual)''':
*** {{mdone|}} Smoke, Regression and Exploratory testing on '''Graphics compatibility & support''' and '''Printing''' → [https://testrail.stage.mozaws.net/index.php?/plans/view/17467 TestRail]
*** {{mdone|}} [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit#gid=1782626591 Bug fix verification]
** '''Update tests (manual)''':
*** {{mdone|}} aurora-cdntest → [https://docs.google.com/document/d/12Vgn5CAyPyCAm64Z5IYGwfdOmdKJtGEJeKfxcObEYes/edit report]
*** {{mdone|}} aurora → [https://docs.google.com/document/d/12Vgn5CAyPyCAm64Z5IYGwfdOmdKJtGEJeKfxcObEYes/edit report]
</div></div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">DevEd 68.0b2</div>
<div class="mw-collapsible-content">
* '''Build''': 68.0b2-build1 ([https://hg.mozilla.org/releases/mozilla-beta/rev/2265bfc5920d 2265bfc5920d], 20190516152822) - <font style="color:green; font-weight:bold;">GREEN</font>
** '''Functional tests (manual)''':
*** {{mdone|}} Smoke, Regression and Exploratory testing on '''Media playback & support''' and '''Crash reporting''' → [https://testrail.stage.mozaws.net/index.php?/plans/view/17528 TestRail]
*** {{mdone|}} [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit#gid=1782626591 Bug fix verification]
** '''Update tests (manual)''':
*** {{mdone|}} aurora-cdntest → [https://docs.google.com/document/d/1eFCpIeC0iVvMosGZ-Iinqfv9oXR_KkbXvDpcNQhO3iM/edit report]
*** {{mdone|}} aurora → [https://docs.google.com/document/d/1eFCpIeC0iVvMosGZ-Iinqfv9oXR_KkbXvDpcNQhO3iM/edit report]
</div></div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">DevEd 68.0b5</div>
<div class="mw-collapsible-content">
* '''Build''': 68.0b5-build1 ([https://hg.mozilla.org/releases/mozilla-beta/rev/7f83ca2147b8a9f 7f83ca2147b8a9f], 20190527103257) - <font style="color:green; font-weight:bold;">GREEN</font>
** '''Functional tests (manual)''':
*** {{mdone|}} Developer Edition Recurring Sign off → [https://testrail.stage.mozaws.net/index.php?/plans/view/17759 TestRail]
*** {{mdone|}} [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit#gid=271991687 Bug fix verification]
** '''Update tests (manual)''':
*** {{mdone|}} aurora-cdntest (as part of Beta 68.0b5 update tests)
*** {{mdone|}} aurora (as part of Beta 68.0b5 update tests)
</div></div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">DevEd 68.0b9</div>
<div class="mw-collapsible-content">
* '''Build''': 68.0b9-build1 ([https://hg.mozilla.org/releases/mozilla-beta/rev/927b23f07645dd3 927b23f07645dd3], 20190610153228) - <font style="color:green; font-weight:bold;">GREEN</font>
** '''Functional tests (manual)''':
*** {{mdone|}} Developer Edition Recurring Sign off → [https://testrail.stage.mozaws.net/index.php?/plans/view/18100 TestRail]
*** {{mdone|}} [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit#gid=271991687 Bug fix verification]
** '''Update tests (manual)''':
*** {{mdone|}} aurora-cdntest  (as part of Beta 68.0b9 update tests)
*** {{mdone|}} aurora (as part of Beta 68.0b9 update tests)
</div></div>
=== Beta build readiness ===
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">Beta 68.0b3</div>
<div class="mw-collapsible-content">
* Build: 68.0b3-build1 ([https://hg.mozilla.org/releases/mozilla-beta/rev/847755a7c325 847755a7c325], 20190520131124) - <font style="color:green; font-weight:bold;">GREEN</font>
** '''Functional tests (manual)''':
*** {{mdone|}} Smoke, Regression and Exploratory testing on '''Plugin compatibility & support''', '''Search''' and '''Awesomebar & indicators''' → [https://testrail.stage.mozaws.net/index.php?/plans/view/17599 TestRail]
*** {{mdone|}} Smoke testing on Ubuntu snap package → [https://testrail.stage.mozaws.net/index.php?/plans/view/17619 TestRail]
*** {{mdone|}} Smoke testing on DLL interceptor → [https://docs.google.com/document/d/1To6OY_hzDaEsOQkR_9VZN9S3dUZkQq_HRQJXLvHaPgQ/edit#heading=h.f8uurjuuffzj Test Results]
*** {{mdone|}} Bug fix verification – [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit?pref=2&pli=1#gid=271991687 Bug list]
** '''Update tests (manual)''':
*** {{mdone|}} beta-cdntest, aurora-cdntest → [https://docs.google.com/document/d/1fxx2H6v22cHVD6KIpAJUUFgIqjVUZDudTFLAsdaQpjI/edit# report]
*** {{mdone|}} beta, aurora → [https://docs.google.com/document/d/1fxx2H6v22cHVD6KIpAJUUFgIqjVUZDudTFLAsdaQpjI/edit# report]
</div></div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">Beta 68.0b4</div>
<div class="mw-collapsible-content">
* Build: 68.0b4-build1 ([https://hg.mozilla.org/releases/mozilla-beta/rev/ddc0c7609e57ad43e1a28888ffffcbdf02131a53 ddc0c7609e57ad43e1a28888ffffcbdf02131a53], 20190523115432) - <font style="color:green; font-weight:bold;">GREEN</font>
** '''Functional tests (manual)''':
*** {{mdone|}} Smoke, Regression and Exploratory testing on '''Build installation & uninstallation''', '''Download manager''', '''Content scroll, zoom & drag support''' and '''Geolocation''' → [https://testrail.stage.mozaws.net/index.php?/reports/view/1679 TestRail]
*** {{mdone|}} Smoke testing on Ubuntu snap package → [https://testrail.stage.mozaws.net/index.php?/reports/view/1677 report]
*** {{mdone|}} Bug fix verification – [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=1553481%2C%201553031%2C%201553029%2C%201553028%2C%201552719%2C%201550697%2C%20416771&list_id=14729987 Bug list]
** '''Update tests (manual)''':
*** {{mdone|}} beta-cdntest, aurora-cdntest → [https://docs.google.com/document/d/17-hF6lQc-dOglhnVJo3xujobTsTdXtl3aBO37xPfVpw/edit report]
*** {{mdone|}} beta, aurora → [https://docs.google.com/document/d/17-hF6lQc-dOglhnVJo3xujobTsTdXtl3aBO37xPfVpw/edit report]
</div></div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">Beta 68.0b5</div>
<div class="mw-collapsible-content">
* Build: 68.0b5-build1 ([https://hg.mozilla.org/releases/mozilla-beta/rev/7f83ca2147b8 7f83ca2147b8], 20190527103257) - <font style="color:green; font-weight:bold;">GREEN</font>
** '''Functional tests (manual)''':
*** {{mdone|}} Smoke, Regression and Exploratory testing on '''Session restore''' and '''Bookmarks''' → [https://testrail.stage.mozaws.net/index.php?/plans/view/17737 TestRail]
*** {{mdone|}} Smoke testing on Ubuntu snap package → [https://testrail.stage.mozaws.net/index.php?/plans/view/17756 report]
*** {{mdone|}} Bug fix verification – [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit?pref=2&pli=1#gid=271991687 Bug List]
** '''Update tests (manual)''':
*** {{mdone|}} beta-cdntest, aurora-cdntest → [https://docs.google.com/document/d/19mbW4Ni__6t9uJz9x4oZWa4KYbIq6s6j6sYM6xS6ozE/edit report]
*** {{mdone|}} beta, aurora → [https://docs.google.com/document/d/19mbW4Ni__6t9uJz9x4oZWa4KYbIq6s6j6sYM6xS6ozE/edit report]
</div></div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">Beta 68.0b6</div>
<div class="mw-collapsible-content">
* Build: 68.0b6-build1 ([https://hg.mozilla.org/releases/mozilla-beta/rev/104b25c67fb01d01383b86a77b5e0cd89e3b1a7f 104b25c67fb01d01383b86a77b5e0cd89e3b1a7f], 20190529145824) - <font style="color:green; font-weight:bold;">GREEN</font>
** '''Functional tests (manual)''':
*** {{mdone|}} Smoke, Regression and Exploratory testing on '''Safe browsing''' and '''Image compatibility & support''' → [https://testrail.stage.mozaws.net/index.php?/reports/view/1693 TestRail]
*** {{mdone|}} Smoke testing on Ubuntu snap package → [https://testrail.stage.mozaws.net/index.php?/reports/view/1692 report]
*** {{mdone|}} Bug fix verification – [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit?pref=2&pli=1#gid=271991687 Bug List]
** '''Update tests (manual)''':
*** {{mdone|}} beta-cdntest, aurora-cdntest → [https://docs.google.com/document/d/1HmV9c9-o0BT753bz1s3vSkYKQ_Jz4ARFEFlmaANpS7o/edit report]
*** {{mdone|}} beta, aurora → [https://docs.google.com/document/d/1HmV9c9-o0BT753bz1s3vSkYKQ_Jz4ARFEFlmaANpS7o/edit report]
</div></div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">Beta 68.0b7</div>
<div class="mw-collapsible-content">
* Build: 68.0b7-build1 ([https://hg.mozilla.org/releases/mozilla-beta/rev/ed47966f79228df65b6326979609fbee94731ef0 ed47966f79228df65b6326979609fbee94731ef0], 20190603181408) - <font style="color:green; font-weight:bold;">GREEN</font>
** '''Functional tests (manual)''':
*** {{mdone|}} Smoke, Regression and Exploratory testing on '''Graphics compatibility & support''', '''WebExtensions compatibility & support''' and '''In-browser PDF support''' → [https://testrail.stage.mozaws.net/index.php?/plans/view/17899 TestRail]
*** {{mdone|}} Smoke testing on Ubuntu snap package → [https://testrail.stage.mozaws.net/index.php?/plans/view/17923 report]
*** {{mdone|}} Bug fix verification – [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit?pref=2&pli=1#gid=271991687 Bug List]
** '''Update tests (manual)''':
*** {{mdone|}} beta-cdntest, aurora-cdntest → [https://docs.google.com/document/d/1LDOlEpLR2zvDCsJw2B_EG1frEZwOV69GTGXGReHpNhg/edit report]
*** {{mdone|}} beta, aurora → [https://docs.google.com/document/d/1LDOlEpLR2zvDCsJw2B_EG1frEZwOV69GTGXGReHpNhg/edit report]
</div></div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">Beta 68.0b8</div>
<div class="mw-collapsible-content">
* Build: 68.0b8-build1 ([https://hg.mozilla.org/releases/mozilla-beta/rev/b25d2ea5cf47fc4e0c01214acad7c30162b1b1d8 b25d2ea5cf47fc4e0c01214acad7c30162b1b1d8], 20190606101422) - <font style="color:green; font-weight:bold;">GREEN</font>
** '''Functional tests (manual)''':
*** {{mdone|}} Smoke, Regression and Exploratory testing on '''Default & custom theme support ''', '''History''' and '''File system interaction''' → [https://testrail.stage.mozaws.net/index.php?/plans/view/17954 TestRail]
*** {{mdone|}} Smoke testing on Ubuntu snap package → [https://testrail.stage.mozaws.net/index.php?/reports/view/1707 report]
*** {{mdone|}} Bug fix verification – [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit?pref=2&pli=1#gid=271991687 Bug List]
** '''Update tests (manual)''':
*** {{mdone|}} beta-cdntest, aurora-cdntest → [https://docs.google.com/document/d/17jTb5qhrTx39KxFK2KyElb2I1M1lduxBVrQloL1rkBI/edit report]
*** {{mdone|}} beta, aurora → [https://docs.google.com/document/d/17jTb5qhrTx39KxFK2KyElb2I1M1lduxBVrQloL1rkBI/edit report]
</div></div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">Beta 68.0b9</div>
<div class="mw-collapsible-content">
* Build: 68.0b9-build1 ([https://hg.mozilla.org/releases/mozilla-beta/rev/927b23f07645dd3f1ff4b63d8cf477bec77b9e22 927b23f07645], 20190610153228) - <font style="color:green; font-weight:bold;">GREEN</font>
** '''Functional tests (manual)''':
*** {{mdone|}} Smoke, Regression and Exploratory testing on '''Private Browsing ''', '''Privacy''' and '''Form Autofill''' → [https://testrail.stage.mozaws.net/index.php?/plans/view/18074 TestRail]
*** {{mdone|}} Smoke testing on Ubuntu snap package → [https://testrail.stage.mozaws.net/index.php?/reports/view/1719 report]
*** {{mdone|}} Bug fix verification – [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit?pref=2&pli=1#gid=271991687 Bug List]
** '''Update tests (manual)''':
*** {{mdone|}} beta-cdntest, aurora-cdntest → [https://docs.google.com/document/d/17jTb5qhrTx39KxFK2KyElb2I1M1lduxBVrQloL1rkBI/edit report]
*** {{mdone|}} beta, aurora → [https://docs.google.com/document/d/17jTb5qhrTx39KxFK2KyElb2I1M1lduxBVrQloL1rkBI/edit report]
</div></div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">Beta 68.0b10</div>
<div class="mw-collapsible-content">
* Build: 68.0b10-build1 ([https://hg.mozilla.org/releases/mozilla-beta/rev/4c434d19d03d 4c434d19d03d], 20190613141208) - <font style="color:green; font-weight:bold;">GREEN</font>
** '''Functional tests (manual)''':
*** {{mdone|}} Smoke, Regression and Exploratory testing on '''WebExtensions compatibility & support''', '''Search''' and '''Profile support & compatibility''' → [https://testrail.stage.mozaws.net/index.php?/plans/view/18165 TestRail]
*** {{mdone|}} Smoke testing on Ubuntu snap package → [https://testrail.stage.mozaws.net/index.php?/plans/view/18187 report]
*** {{mdone|}} Bug fix verification – [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit?pref=2&pli=1#gid=271991687 Bug List]
** '''Update tests (manual)''':
*** {{mdone|}} beta-cdntest, aurora-cdntest → [https://docs.google.com/document/d/1nHr4G9FtrcqCOXhpdaL3E_kWcf7Seyko2fmOJBQcEGg/edit report]
*** {{mdone|}} beta, aurora → [https://docs.google.com/document/d/1nHr4G9FtrcqCOXhpdaL3E_kWcf7Seyko2fmOJBQcEGg/edit report]
</div></div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">Beta 68.0b11</div>
<div class="mw-collapsible-content">
* Build: 68.0b11-build1 ([https://hg.mozilla.org/releases/mozilla-beta/rev/60f0c79f0f53fcbe 60f0c79f0f53fcbe], 20190617172838) - <font style="color:green; font-weight:bold;">GREEN</font>
** '''Functional tests (manual)''':
*** {{mdone|}} Smoke, Regression and Exploratory testing on '''Data Migration''' and '''Toolbars & window controls''' → [https://testrail.stage.mozaws.net/index.php?/plans/view/18218 TestRail]
*** {{mdone|}} Smoke testing on Ubuntu snap package → [https://testrail.stage.mozaws.net/index.php?/plans/view/18234 report]
*** {{mdone|}} Bug fix verification – [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit?pref=2&pli=1#gid=271991687 Bug List]
** '''Update tests (manual)''':
*** {{mdone|}} beta-cdntest, aurora-cdntest → [https://docs.google.com/document/d/1PrJvP60wvSvyyA04Bf7C0DtM-g_rojkYdHb7cLFUSuQ/edit report]
*** {{mnotstarted|}} beta, aurora → [https://docs.google.com/document/d/1PrJvP60wvSvyyA04Bf7C0DtM-g_rojkYdHb7cLFUSuQ/edit report]
</div></div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">Beta 68.0b12</div>
<div class="mw-collapsible-content">
* Build: 68.0b12-build1 ([https://hg.mozilla.org/releases/mozilla-beta/rev/046390a25abd 046390a25abd], 20190619234730) - <font style="color:green; font-weight:bold;">GREEN</font>
** '''Functional tests (manual)''':
*** {{mdone|}} Smoke, Regression and Exploratory testing on '''Crash reporting''', '''Plugin compatibility & support'''and '''Geolocation''' → [https://testrail.stage.mozaws.net/index.php?/plans/view/18292 TestRail]
*** {{mdone|}} Smoke testing on Ubuntu snap package → [https://testrail.stage.mozaws.net/index.php?/plans/view/18293 report]
*** {{mdone|}} Bug fix verification – [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit?pref=2&pli=1#gid=271991687 Bug List]
** '''Update tests (manual)''':
*** {{mdone|}} beta-cdntest, aurora-cdntest → [https://docs.google.com/document/d/1dkGW5enod_IXRJxhqrIm06IAkVpwPMrhtjTvNakY24Y/edit report]
*** {{mdone|}} beta, aurora → [https://docs.google.com/document/d/1dkGW5enod_IXRJxhqrIm06IAkVpwPMrhtjTvNakY24Y/edit report]
</div></div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">Beta 68.0b13</div>
<div class="mw-collapsible-content">
* Build: 68.0b13-build1 ([https://hg.mozilla.org/releases/mozilla-beta/rev/da9f01df3bef da9f01df3bef], 20190624133534) - <font style="color:green; font-weight:bold;">GREEN</font>
** '''Functional tests (manual)''':
*** {{mdone|}} Smoke, Regression and Exploratory testing on '''Session restore'''and '''Content scroll, zoom & drag support''' → [https://testrail.stage.mozaws.net/index.php?/plans/view/18452 TestRail]
*** {{mdone|}} Smoke testing on Ubuntu snap package → [https://testrail.stage.mozaws.net/index.php?/plans/view/18474 report]
*** {{mdone|}} Bug fix verification – [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit?pref=2&pli=1#gid=271991687 Bug List]
** '''Update tests (manual)''':
*** {{mdone|}} beta-cdntest, aurora-cdntest → [https://docs.google.com/document/d/1EJ3GgYVhTGIBdVTt5XI58OfsUDktO05Y0cUrxm87RCk/edit report]
*** {{mdone|}} beta, aurora → [https://docs.google.com/document/d/1EJ3GgYVhTGIBdVTt5XI58OfsUDktO05Y0cUrxm87RCk/edit report]
</div></div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">Beta 68.0b14</div>
<div class="mw-collapsible-content">
* Build: 68.0b14-build1 ([https://hg.mozilla.org/releases/mozilla-beta/rev/a38e711fb744 a38e711fb744], 20190627143605) - <font style="color:green; font-weight:bold;">GREEN</font>
** '''Functional tests (manual)''':
*** {{mdone|}} Smoke, Regression and Exploratory testing on '''Bookmarks''' and '''Media playback & support''' → [https://testrail.stage.mozaws.net/index.php?/plans/view/18522 TestRail]
*** {{mdone|}} Smoke testing on Ubuntu snap package → [https://testrail.stage.mozaws.net/index.php?/plans/view/18537 report]
*** {{mdone|}} Bug fix verification – [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit?pref=2&pli=1#gid=271991687 Bug List]
** '''Update tests (manual)''':
*** {{mdone|}} beta-cdntest, aurora-cdntest → [https://docs.google.com/document/d/1gsRcOLCZ6sSYf0TJPOmC85S_nFoDCMxHjcP3pLIibFU/edit# report]
*** {{mnotstarted|}} beta, aurora → [https://docs.google.com/document/d/1gsRcOLCZ6sSYf0TJPOmC85S_nFoDCMxHjcP3pLIibFU/edit# report]
</div></div>
== Release 68 milestones ==
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">RC1 68.0</div>
<div class="mw-collapsible-content">
* '''Build''': 68.0-build1 ([https://hg.mozilla.org/releases/mozilla-beta/rev/3fc582b1f4b2 3fc582b1f4b2], 20190701181138) - <font style="color:green; font-weight:bold;">GREEN</font>
** '''Functional tests (manual)''':
*** {{mdone|}} Smoke, Regression and Exploratory testing on '''Web compatibility''' and '''WebExtensions compatibility & support''' → [https://testrail.stage.mozaws.net/index.php?/plans/view/18586 TestRail]
*** {{mdone|}} Smoke testing on Ubuntu snap package → [https://testrail.stage.mozaws.net/index.php?/plans/view/18608 report]
*** {{mdone|}} Smoke testing on Windows Acer builds → [https://docs.google.com/document/d/1n4Kg6mvy9GBr0KxUVvYnd-LP4S6bHcYHE8KajXuS4UM/edit# report]
*** {{mprog|}} Bug fix verification – [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit#gid=271991687 Bug verification]
** '''Update tests (manual)''':
*** {{mdone|}} beta-cdntest → [https://docs.google.com/document/d/1nbK0ScU8PekxQLrJ99Fz2R3OJNuUT-K4dSuz26AYKi4/edit report]
*** {{mnotstarted|}} beta → skipped
*** {{mnotstarted|}} release-localtest → skipped
*** {{mnotstarted|}} release-cdntest → skipped
*** {{mnotstarted|}} release → skipped
</div></div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">RC2 68.0</div>
<div class="mw-collapsible-content">
* '''Build''': 68.0-build2 ([https://hg.mozilla.org/releases/mozilla-beta/rev/3a52b315fac0 3a52b315fac0], 20190704163141) - <font style="color:green; font-weight:bold;">GREEN</font>
** '''Functional tests (manual)''':
*** {{mdone|}} Smoke, Regression and Exploratory testing on '''Web compatibility''' → [https://testrail.stage.mozaws.net/index.php?/plans/view/18753 TestRail]
*** {{mdone|}} Smoke testing on Ubuntu snap package → [https://testrail.stage.mozaws.net/index.php?/plans/view/18774 report]
*** {{mdone|}} Smoke testing on Windows Acer builds → [https://docs.google.com/document/d/17D7JnBOj5tEFdfavEafWaliSLQxybCDbj4NAbgC_qGs/edit#heading=h.ra59jc33u21r report]
*** {{mdone|}} Bug fix verification – [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit#gid=271991687 Bug verification]
** '''Update tests (manual)''':
*** {{mnotstarted|}} beta-cdntest → [https://drive.google.com/open?id=1RotoD7oZcSYC9hOcdnTP_-S8RTLoDgiz3qI7OnKjOZY report]
*** {{mnotstarted|}} beta → skipped
*** {{mnotstarted|}} release-localtest → skipped
*** {{mnotstarted|}} release-cdntest → skipped
*** {{mnotstarted|}} release → skipped
</div></div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">RC3 68.0</div>
<div class="mw-collapsible-content">
* '''Build''': 68.0-build3 ([https://hg.mozilla.org/releases/mozilla-beta/rev/353628fec415 353628fec415], 20190705220548) - <font style="color:green; font-weight:bold;">GREEN</font>
** '''Functional tests (manual)''':
*** {{mdone|}} Smoke, Regression and Exploratory testing on '''Web compatibility''' → [https://testrail.stage.mozaws.net/index.php?/plans/view/18789 TestRail]
*** {{mdone|}} Smoke testing on Ubuntu snap package → [https://testrail.stage.mozaws.net/index.php?/reports/view/1832 report]
*** {{mdone|}} Smoke testing on Windows Acer builds → [https://docs.google.com/document/d/17D7JnBOj5tEFdfavEafWaliSLQxybCDbj4NAbgC_qGs/edit#heading=h.ra59jc33u21r report]
*** {{mdone|}} Bug fix verification – [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit#gid=271991687 Bug verification]
** '''Update tests (manual)''':
*** {{mdone|}} beta-cdntest → [https://docs.google.com/document/d/1PtwR_oXQzg_HWhkFNLbFfTPL_-crCa__IPgHtUQoge8/edit# report]
*** {{mdone|}} beta → [https://docs.google.com/document/d/1PtwR_oXQzg_HWhkFNLbFfTPL_-crCa__IPgHtUQoge8/edit# report]
*** {{mnotstarted|}} release-localtest → skipped
*** {{mdone|}} release-cdntest → [https://docs.google.com/document/d/1AuH79AthcY2DiitN4IQWYqK-VJxlZP_kN5yTWVEX0v0/edit report]
*** {{mnotstarted|}} release → report
</div></div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">RC 68.0.1</div>
<div class="mw-collapsible-content">
* '''Build''': 68.0.1-build1 ([https://hg.mozilla.org/releases/mozilla-release/rev/837bbcb850cd 837bbcb850cd], 20190717172542) - <font style="color:green; font-weight:bold;">GREEN</font>
** '''Functional tests (manual)''':
*** {{mdone|}} Smoke, Regression and Exploratory testing on '''Web compatibility''' → [https://testrail.stage.mozaws.net/index.php?/reports/view/1848 TestRail]
*** {{mdone|}} Smoke testing on Ubuntu snap package → [https://testrail.stage.mozaws.net/index.php?/reports/view/1853 report]
*** {{mdone|}} Smoke testing on Windows Acer builds → [https://docs.google.com/document/d/1nfUcNS0WtDoVRVb5SCA_a_q3ZgOpwiivJ_AeMnbGcIE/edit# report]
*** {{mdone|}} Bug fix verification – [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=1553586%2C1556846%2C1558503%2C1565779%2C1566206&list_id=14810958 Bug verification]
** '''Update tests (manual)''':
*** {{mdone|}} release-cdntest → [https://docs.google.com/document/d/1x62rHtXpS9q6XS6Ft8iWgCU-wMIHthloWt-20Kn4Wz8/edit report]
*** {{mnotstarted|}} release → [https://docs.google.com/document/d/1x62rHtXpS9q6XS6Ft8iWgCU-wMIHthloWt-20Kn4Wz8/edit report]
</div></div>
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">RC 68.0.2</div>
<div class="mw-collapsible-content">
* '''Build''': 68.0.2-build1 ([https://hg.mozilla.org/releases/mozilla-release/rev/7ece03f69719 7ece03f69719], 20190813150448) - <font style="color:green; font-weight:bold;">GREEN</font>
** '''Functional tests (manual)''':
*** {{mdone|}} Smoke, Regression and Exploratory testing on '''Web compatibility''' → [https://testrail.stage.mozaws.net/index.php?/plans/view/19576 TestRail]
*** {{mdone|}} Smoke testing on Ubuntu snap package → [https://testrail.stage.mozaws.net/index.php?/reports/view/1922 report]
*** {{mdone|}} Smoke testing on Windows Acer builds → [https://docs.google.com/document/d/1C-_C19mjcl351jygT5Dyo64AT2zFWlu1oHsQoEOrBU4/edit report]
*** {{mdone|}} Bug fix verification – [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=1560228%2C%201565542%2C%201567614%2C%201569244&list_id=14850527 Bug verification]
** '''Update tests (manual)''':
*** {{mdone|}} release-cdntest → [https://docs.google.com/spreadsheets/d/1aLXxd7yMJaisaWPaW_lBTeMyOyGXiie-pRjoJrphu4E/edit#gid=850719639 report]
*** {{mnotstarted|}} release → [https://docs.google.com/spreadsheets/d/1aLXxd7yMJaisaWPaW_lBTeMyOyGXiie-pRjoJrphu4E/edit#gid=850719639 report]
</div></div>
</div></div>

Latest revision as of 13:51, 14 August 2019

Summary

The purpose of this page is to track the testing of Firefox 68 from Nightly to Release, and post Release if needed.

Release schedule

Status meetings

The Channel Status meeting is used to raise quality concerns with the Release Management team.

QA points of contact

Communication channels

Firefox for Desktop

Engineering QA
  • Reporting to: Tom Grabowski, Tania Maity
  • Softvision Manager: Rares Bologa
  • Team Leads: Brindusa Tot
  • Offices: Romania (EEST), Argentina (GMT-3)
  • Responsible for: new feature testing, Nightly build readiness, Nightly build regression testing, bug triage, bug verification, bug investigation
Release QA
  • Reporting to: Tom Grabowski, Tania Maity
  • Softvision Manager: Andrei Vaida
  • Team Leads: Bogdan Maris, Cornel Ionce
  • Office: Romania (EEST)
  • Responsible for: new feature testing, Beta build readiness, Developer Edition build readiness, ESR build readiness, RC build readiness, Dot release build readiness, Chemspill build readiness, bug verification, bug investigation

Firefox for Mobile

Release QA
Firefox for iOS QA

AMO, WebExtensions

Add-ons QA
  • Reporting to: Krupa Raj
  • Softvision Manager: Rares Bologa
  • Team Leads: Victor Carciu, Madalin Cotetiu
  • Office: Romania (EEST)
  • Responsible for: AMO feature testing, AMO weekly release, AMO Sanity, AMO bug verification, Webextensions features, Webextensions bug triage and verification, Telemetry

Experiments, Normandy, Subscription services, and other Firefox features

Experiments QA
  • Reporting to: Krupa Raj
  • Softvision Manager: Rares Bologa
  • Team Leads: Paul Oiegas
  • Office: Romania (EEST)
  • Responsible for: testing Experiments, Normandy, Experimenter website, Activity Stream, Screenshots, Send (Web & Android) and Lockbox Desktop.

QA activities

New feature readiness

All new features tested by QA and shipping with Firefox 68 are available here.

The feature testing process followed by QA is available here.

Bug fix verification

  • Daily verification of uplifted bugs - these are uplifts, triaged daily and verified under ~48hrs
  • Ongoing verification of Nightly bugs - this is the general bug verification done throughout the Nightly cycle; it sometimes overlaps with uplifts verification; also involves active monitoring of qe-verify? bugs
  • Ongoing verification of Beta bugs - this is the general bug verification done throughout the Beta cycle; it sometimes overlaps with uplifts verification; also involves active monitoring of qe-verify? bugs
  • On-demand verification of bugs - these are special requests (ideally coming in through PI, but also through Bugzilla needinfo?, chat or emails)
  • Periodical verification of security bugs - these are post-critsmash bugs, triaged and verified ~every 2wks

Unconfirmed bug triage

This is the triage conducted by Engineering QA.

Security bug triage

This is the post-CritSmash triage conducted by Engineering QA (on Nightly) and Release QA (on Beta, Release, ESR).

Bug investigation

This is the daily check and investigation of new regressionwindow-wanted bugs, conducted by Engineering QA and Release QA.

Nightly 68 milestones

[early/mid cycle] Nightly build readiness

Scope: assess build readiness for shipping to Beta by testing Nightly periodically and providing a list of issues (if any) that need to be addressed before the merge (more on this).

Nightly 68.0a1 (2019-04-04)
  • Build: 68.0a1 (commit, ID) - GREEN
    • [DONE] Functional testing (manual) → report
      • New bugs
        • Regressions: 1541400
        • Non-regressions: 1541376, 1541345, 1541057
      • Known bugs
        • High-impact: none
        • Low/medium-impact: bug list
Nightly 68.0a1 (2019-04-18)
  • Build: 68.0a1 (commit, ID) - GREEN
    • [DONE] Functional testing (manual) → report
      • New bugs
        • Regressions: 1544509
        • Non-regressions: 1544701
      • Known bugs
        • High-impact: none
        • Low/medium-impact: bug list

[prior to merge] Nightly regression testing

Nightly 68.0a1 (2019-05-16)
  • Build: 68.0a1 (commit, ID) - GREEN
    • [DONE] Regression testing (manual) → report
      • New bugs
        • Regressions: 1550523, 1551554, 1551215, 1551452
        • Non-regressions: 1544509, 1549360, 1550459, 1550533, 1551534
      • Known bugs
        • High-impact: none
        • Low/medium-impact: bug list

[optional] [prior to merge] Nightly new feature smoke testing

Note: this is optional, based on available QA bandwidth.

Beta/DevEd 68 milestones

Developer Edition build readiness

Scope: assess build stability periodically (around every 2 weeks) to make sure we are shipping quality products to our Developer Edition users from the aurora update channel (more on this).

DevEd 68.0b1
  • Build: 68.0b1-build1 (d313e24f18c2, 20190513111358) - GREEN
    • Functional tests (manual):
    • Update tests (manual):
DevEd 68.0b2
  • Build: 68.0b2-build1 (2265bfc5920d, 20190516152822) - GREEN
    • Functional tests (manual):
    • Update tests (manual):
DevEd 68.0b5
  • Build: 68.0b5-build1 (7f83ca2147b8a9f, 20190527103257) - GREEN
    • Functional tests (manual):
    • Update tests (manual):
      • [DONE] aurora-cdntest (as part of Beta 68.0b5 update tests)
      • [DONE] aurora (as part of Beta 68.0b5 update tests)
DevEd 68.0b9
  • Build: 68.0b9-build1 (927b23f07645dd3, 20190610153228) - GREEN
    • Functional tests (manual):
    • Update tests (manual):
      • [DONE] aurora-cdntest (as part of Beta 68.0b9 update tests)
      • [DONE] aurora (as part of Beta 68.0b9 update tests)

Beta build readiness

Beta 68.0b3
  • Build: 68.0b3-build1 (847755a7c325, 20190520131124) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Plugin compatibility & support, Search and Awesomebar & indicators → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → TestRail
      • [DONE] Smoke testing on DLL interceptor → Test Results
      • [DONE] Bug fix verification – Bug list
    • Update tests (manual):
      • [DONE] beta-cdntest, aurora-cdntest → report
      • [DONE] beta, aurora → report
Beta 68.0b4
  • Build: 68.0b4-build1 (ddc0c7609e57ad43e1a28888ffffcbdf02131a53, 20190523115432) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Build installation & uninstallation, Download manager, Content scroll, zoom & drag support and Geolocation → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → report
      • [DONE] Bug fix verification – Bug list
    • Update tests (manual):
      • [DONE] beta-cdntest, aurora-cdntest → report
      • [DONE] beta, aurora → report
Beta 68.0b5
  • Build: 68.0b5-build1 (7f83ca2147b8, 20190527103257) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Session restore and Bookmarks → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → report
      • [DONE] Bug fix verification – Bug List
    • Update tests (manual):
      • [DONE] beta-cdntest, aurora-cdntest → report
      • [DONE] beta, aurora → report
Beta 68.0b6
  • Build: 68.0b6-build1 (104b25c67fb01d01383b86a77b5e0cd89e3b1a7f, 20190529145824) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Safe browsing and Image compatibility & support → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → report
      • [DONE] Bug fix verification – Bug List
    • Update tests (manual):
      • [DONE] beta-cdntest, aurora-cdntest → report
      • [DONE] beta, aurora → report
Beta 68.0b7
  • Build: 68.0b7-build1 (ed47966f79228df65b6326979609fbee94731ef0, 20190603181408) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Graphics compatibility & support, WebExtensions compatibility & support and In-browser PDF support → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → report
      • [DONE] Bug fix verification – Bug List
    • Update tests (manual):
      • [DONE] beta-cdntest, aurora-cdntest → report
      • [DONE] beta, aurora → report
Beta 68.0b8
  • Build: 68.0b8-build1 (b25d2ea5cf47fc4e0c01214acad7c30162b1b1d8, 20190606101422) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Default & custom theme support , History and File system interaction → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → report
      • [DONE] Bug fix verification – Bug List
    • Update tests (manual):
      • [DONE] beta-cdntest, aurora-cdntest → report
      • [DONE] beta, aurora → report
Beta 68.0b9
  • Build: 68.0b9-build1 (927b23f07645, 20190610153228) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Private Browsing , Privacy and Form Autofill → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → report
      • [DONE] Bug fix verification – Bug List
    • Update tests (manual):
      • [DONE] beta-cdntest, aurora-cdntest → report
      • [DONE] beta, aurora → report
Beta 68.0b10
  • Build: 68.0b10-build1 (4c434d19d03d, 20190613141208) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on WebExtensions compatibility & support, Search and Profile support & compatibility → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → report
      • [DONE] Bug fix verification – Bug List
    • Update tests (manual):
      • [DONE] beta-cdntest, aurora-cdntest → report
      • [DONE] beta, aurora → report
Beta 68.0b11
  • Build: 68.0b11-build1 (60f0c79f0f53fcbe, 20190617172838) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Data Migration and Toolbars & window controls → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → report
      • [DONE] Bug fix verification – Bug List
    • Update tests (manual):
      • [DONE] beta-cdntest, aurora-cdntest → report
      • [NOT STARTED] beta, aurora → report
Beta 68.0b12
  • Build: 68.0b12-build1 (046390a25abd, 20190619234730) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Crash reporting, Plugin compatibility & supportand Geolocation → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → report
      • [DONE] Bug fix verification – Bug List
    • Update tests (manual):
      • [DONE] beta-cdntest, aurora-cdntest → report
      • [DONE] beta, aurora → report
Beta 68.0b13
  • Build: 68.0b13-build1 (da9f01df3bef, 20190624133534) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Session restoreand Content scroll, zoom & drag support → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → report
      • [DONE] Bug fix verification – Bug List
    • Update tests (manual):
      • [DONE] beta-cdntest, aurora-cdntest → report
      • [DONE] beta, aurora → report
Beta 68.0b14
  • Build: 68.0b14-build1 (a38e711fb744, 20190627143605) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Bookmarks and Media playback & support → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → report
      • [DONE] Bug fix verification – Bug List
    • Update tests (manual):
      • [DONE] beta-cdntest, aurora-cdntest → report
      • [NOT STARTED] beta, aurora → report

Release 68 milestones

RC1 68.0
  • Build: 68.0-build1 (3fc582b1f4b2, 20190701181138) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Web compatibility and WebExtensions compatibility & support → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → report
      • [DONE] Smoke testing on Windows Acer builds → report
      • [IN PROGRESS] Bug fix verification – Bug verification
    • Update tests (manual):
      • [DONE] beta-cdntest → report
      • [NOT STARTED] beta → skipped
      • [NOT STARTED] release-localtest → skipped
      • [NOT STARTED] release-cdntest → skipped
      • [NOT STARTED] release → skipped
RC2 68.0
  • Build: 68.0-build2 (3a52b315fac0, 20190704163141) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Web compatibility → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → report
      • [DONE] Smoke testing on Windows Acer builds → report
      • [DONE] Bug fix verification – Bug verification
    • Update tests (manual):
      • [NOT STARTED] beta-cdntest → report
      • [NOT STARTED] beta → skipped
      • [NOT STARTED] release-localtest → skipped
      • [NOT STARTED] release-cdntest → skipped
      • [NOT STARTED] release → skipped
RC3 68.0
  • Build: 68.0-build3 (353628fec415, 20190705220548) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Web compatibility → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → report
      • [DONE] Smoke testing on Windows Acer builds → report
      • [DONE] Bug fix verification – Bug verification
    • Update tests (manual):
      • [DONE] beta-cdntest → report
      • [DONE] beta → report
      • [NOT STARTED] release-localtest → skipped
      • [DONE] release-cdntest → report
      • [NOT STARTED] release → report
RC 68.0.1
  • Build: 68.0.1-build1 (837bbcb850cd, 20190717172542) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Web compatibility → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → report
      • [DONE] Smoke testing on Windows Acer builds → report
      • [DONE] Bug fix verification – Bug verification
    • Update tests (manual):
      • [DONE] release-cdntest → report
      • [NOT STARTED] release → report
RC 68.0.2
  • Build: 68.0.2-build1 (7ece03f69719, 20190813150448) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Web compatibility → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → report
      • [DONE] Smoke testing on Windows Acer builds → report
      • [DONE] Bug fix verification – Bug verification
    • Update tests (manual):
      • [DONE] release-cdntest → report
      • [NOT STARTED] release → report
Retrieved from "https://wiki.allizom.org/index.php?title=Releases/Firefox_68/Test_Plan&oldid=1216516"