Releases/Firefox 59/Test Plan: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(-)
 
 
(74 intermediate revisions by 7 users not shown)
Line 4: Line 4:
The QA efforts are mostly split up between two teams:
The QA efforts are mostly split up between two teams:
* '''[[QA_SoftVision_Team#Engineering_QA_Team|Engineering QA]]'''
* '''[[QA_SoftVision_Team#Engineering_QA_Team|Engineering QA]]'''
** team reports to [https://phonebook.mozilla.org/tree.php?search/lmandel@mozilla.com Lawrence Mandel]
** team reports to [https://phonebook.mozilla.org/tree.php?search/rvandermeluen@mozilla.com Ryan VanderMeulen] and [https://phonebook.mozilla.org/tree.php?search/tgrabowski@mozilla.com Tom Grabowski]
** the Project Manager (and main contact) for the team is [https://phonebook.mozilla.org/tree.php?search/rbologa@mozilla.com Rares Bologa]
** the Project Manager (and main contact) for the team is [https://phonebook.mozilla.org/tree.php?search/rbologa@mozilla.com Rares Bologa]
** team is based in Romania (most part) and Las Vegas (5 people)
** team is based in Romania (most part) and Las Vegas (5 people)
Line 172: Line 172:
  ! width="200" | TEST REPORT
  ! width="200" | TEST REPORT
  |-
  |-
  | style="text-align:center;" | date
  | style="text-align:center;" | 2017-11-23
  | style="background:blue; color: white;" | TBD
  | style="background:green; color: white;" | GREEN
  | style="text-align: left;" | tbd
  | style="text-align: left;" |  
  | style="text-align: center;" | tbd
* 1 new, low-impact bug uncovered: {{bug|1419819}}.
  | style="text-align: center;" | tbd
* Known (old) bugs are low-impact.
  | style="text-align: center;" | tbd
  | style="text-align: center;" | ---
  | style="text-align: center;" | [https://mzl.la/2BFGmnG Bugzilla link]
  | style="text-align: center;" | [https://testrail.stage.mozaws.net/index.php?/reports/view/703 TestRail link]
  |-
  |-
  | style="text-align:center;" | date
  | style="text-align:center;" | 2017-12-15
  | style="background:blue; color: white;" | TBD
  | style="background:green; color: white;" | GREEN
  | style="text-align: left;" | tbd
  | style="text-align: left;" |  
  | style="text-align: center;" | tbd
* 3 new bugs uncovered, all of them low-impact: [https://mzl.la/2BGGraX Bugzilla link].
  | style="text-align: center;" | tbd
* Known (old) bugs are low-impact.
  | style="text-align: center;" | tbd
  | style="text-align: center;" | ---
  | style="text-align: center;" | [https://goo.gl/XCm1fH Bugzilla link]
  | style="text-align: center;" | [https://testrail.stage.mozaws.net/index.php?/reports/view/729 TestRail link]
  |-
  |-
  | style="text-align:center;" | date
  | style="text-align:center;" | 2018-01-11
  | style="background:blue; color: white;" | TBD
  | style="background:blue; color: white;" | TBD
  | style="text-align: left;" | tbd
  | style="text-align: left;" | tbd
Line 196: Line 200:
== Nightly Migration ==
== Nightly Migration ==
Pre-sign-off Checks:
Pre-sign-off Checks:
* {{mnotstarted|}} Manual Smoke testing - ensure there are no major issues blocking release to the beta channel
** ''Areas covered: TBD ''
** ''Signed off by <name> on <date>''
*** ''Detailed test results: [https://testrail.stage.mozaws.net/index.php?/plans/view/xxxx in TestRail] (requires access rights) or [https://drive.google.com/open?id=xxxxxxxxxxxx as a pdf].''
* {{mnotstarted|}} Features - all scoped features are signed off as ''ready for Beta'' by owners. (If you haven't heard anything to the contrary, that is probably true, but make sure we don't know of blocking issues.)
* {{mnotstarted|}} Features - all scoped features are signed off as ''ready for Beta'' by owners. (If you haven't heard anything to the contrary, that is probably true, but make sure we don't know of blocking issues.)


Post-sign-off Checks (as soon as updates are enabled again):
Post-sign-off Checks (as soon as updates are enabled again):
* ''None yet for migration of Nightly to Beta''
* ''None yet for migration of Nightly to Beta''
== Developer Edition ==
{{note | Developer Edition builds are usually signed off once every two weeks.}}
{| class="wikitable" width="100%"
! rowspan="2" style="text-align: center;" width=150 | BUILD
! colspan="3" style="text-align: center;" | SIGN OFF
! colspan="2" style="text-align: center;" | ENCOUNTERED BUGS
|-
! style="text-align: center;" width=150 | Status
! style="text-align: center;" width=500 | Reason(s)
! style="text-align: center;" width=80 | Test Report
! style="text-align: center;" width=400 | New
! style="text-align: center;" width=400 | Known (old)
|-
| style="text-align: center;" | 59.0b1-build11
| style="text-align: center; background:green; color:white" | GREEN
|
* No new issues uncovered.
* Known issues are low-impact.
| style="text-align: center;" | [https://testrail.stage.mozaws.net/index.php?/reports/view/769 TestRail link]
| style="text-align: left;" |
* Regressions: none.
* Non-regressions: none.
| style="text-align: left;" |
* High-impact: none.
* Low-impact: none.
|-
| style="text-align: center;" | 59.0b2-build1
| style="text-align: center; background:green; color:white" | GREEN
|
* One new issue uncovered: [https://bugzil.la/1431674 1431674], recent regression, not a blocker.
* Known issues are low-impact.
| style="text-align: center;" | [https://testrail.stage.mozaws.net/index.php?/reports/view/772 TestRail link]
| style="text-align: left;" |
* Regressions: [https://bugzil.la/1431674 1431674].
* Non-regressions: none.
| style="text-align: left;" |
* High-impact: none.
* Low-impact: [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=558178%2C%201309534%2C%201233359%2C%201345877%2C%201350847%2C%201306590%2C%20&list_id=13920689 link].
|-
| style="text-align: center;" | 59.0b5-build1
| style="text-align: center; background:green; color:white" | GREEN
|
* No new issues uncovered during testing
* Known issues are low-impact.
| style="text-align: center;" | [https://testrail.stage.mozaws.net/index.php?/reports/view/779 TestRail link]
| style="text-align: left;" |
* Regressions: none.
* Non-regressions: none.
| style="text-align: left;" |
* High-impact: none.
* Low-impact: [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=1437853%2C%201194214%2C%201309534%2C%201431674%2C%20558178%2C%201429065%2C%201233359%2C%201306590%2C%201345877%2C%201411533%2C%201410009%2C%20428378%2C%201393460%2C%20972658%2C%201306561&list_id=14012856 link].
|-
| style="text-align: center;" | 59.0b9-build1
| style="text-align: center; background:green; color:white" | GREEN
|
* Regressions: none.
* Non-regressions: none.
| style="text-align: center;" | [https://testrail.stage.mozaws.net/index.php?/reports/view/797 TestRail link]
| style="text-align: left;" |
* Regressions: none.
* Non-regressions: none.
| style="text-align: left;" |
* High-impact: none.
* Low-impact: [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=1437853%2C%201194214%2C%201309534%2C%201431674%2C%20558178%2C%201429065%2C%201233359%2C%201306590%2C%201345877%2C%201411533%2C%201410009%2C%20428378%2C%201393460%2C%20972658%2C%201306561&list_id=14012856 list].
|-
| style="text-align: center;" | 59.0b13-build1
| style="text-align: center; background:green; color:white" | GREEN
|
* One new issue uncovered: [https://bugzil.la/1441843 1441843], recent regression, not a blocker.
* Known issues are low-impact.
| style="text-align: center;" | [https://testrail.stage.mozaws.net/index.php?/reports/view/811 TestRail link]
| style="text-align: left;" |
* Regressions: none.
* Non-regressions: none.
| style="text-align: left;" |
* High-impact: none.
* Low-impact: [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=428378%20558178%20972658%201056056%201191290%201194214%201233359%201238955%201306561%201306590%201345877%201373643%201411533%201429065%201431674%201437853%201439536&list_id=14032525 list].
|}
== Beta ==
{{note | Beta builds are usually signed off twice a week.}}
=== Beta 3 ===
==== Build Information ====
* Build 2: [https://archive.mozilla.org/pub/firefox/candidates/59.0b3-candidates/build2/ 59.0b3-build2] | [https://hg.mozilla.org/releases/mozilla-beta/rev/c94e1fd9fe7a c94e1fd9fe7a] | (ID: 20180122192913)
==== Testing ====
* Manual - [https://wiki.mozilla.org/Releases/Firefox_59/Test_Plan/Beta/3 Firefox 59 Beta 3]
** 59.0b3-build2
*** {{mdone|}} Smoke and Exploratory Testing with focus on: Form Autofill V2, PDFium printing on Windows and Mac OS X signing.
**** Detailed test results: [https://testrail.stage.mozaws.net/index.php?/reports/view/tbd in TestRail].
*** {{mdone|}} [https://goo.gl/C8A1e4 Bug verification] with focus on the latest fixes to land in this build.
* Updates testing
** {{mdone|}} Updates on ''beta-cdntest'' channel , [https://public.etherpad-mozilla.org/p/beta-59.0b3-build2 report]
** {{mdone|}} Updates on ''beta'' channel , [https://public.etherpad-mozilla.org/p/beta-59.0b3-build2 report]
=== Beta 4 ===
==== Build Information ====
* Build 1: [https://archive.mozilla.org/pub/firefox/candidates/59.0b4-candidates/build1/ 59.0b4-build1] | [https://hg.mozilla.org/releases/mozilla-beta/rev/783507b1b80d 783507b1b80d] | (ID: 20180125163041)
==== Testing ====
* Manual - [https://wiki.mozilla.org/Releases/Firefox_59/Test_Plan/Beta/4 Firefox 59 Beta 4]
** 59.0b4-build1
*** {{mdone|}} Smoke and Exploratory Testing with focus on: Graphics and Search.
**** Detailed test results: [https://testrail.stage.mozaws.net/index.php?/reports/view/tbd in TestRail].
*** {{mdone|}} [https://goo.gl/C8A1e4 Bug verification] with focus on the latest fixes to land in this build.
* Updates testing
** {{mdone|}} Updates on ''beta-cdntest'' channel , [https://public.etherpad-mozilla.org/p/beta-59.0b4-build1 report]
** {{mdone|}} Updates on ''beta'' channel , [https://public.etherpad-mozilla.org/p/beta-59.0b4-build1 report]
=== Beta 5 ===
==== Build Information ====
* Build 1: [https://archive.mozilla.org/pub/firefox/candidates/59.0b5-candidates/build1/ 59.0b5-build1] | [https://hg.mozilla.org/releases/mozilla-beta/rev/64737c752ac4 64737c752ac4] | (ID: 20180128191456)
==== Testing ====
* Manual - [https://wiki.mozilla.org/Releases/Firefox_59/Test_Plan/Beta/5 Firefox 59 Beta 5]
** 59.0b5-build1
*** {{mdone|}} Smoke and Exploratory Testing with focus on: Web compatibility and Bookmarks.
**** Detailed test results: [https://testrail.stage.mozaws.net/index.php?/reports/view/778 in TestRail].
*** {{mdone|}} [https://goo.gl/C8A1e4 Bug verification] with focus on the latest fixes to land in this build.
* Updates testing
** {{mdone|}} Updates on ''beta-cdntest'' channel , [https://public.etherpad-mozilla.org/p/beta-59.0b5-build1 report]
** {{mdone|}} Updates on ''beta'' channel , [https://public.etherpad-mozilla.org/p/beta-59.0b5-build1 report]
=== Beta 6 ===
==== Build Information ====
* Build 1: [https://archive.mozilla.org/pub/firefox/candidates/59.0b6-candidates/build1/ 59.0b6-build1] | [https://hg.mozilla.org/releases/mozilla-beta/rev/1b4ab9a276ce 1b4ab9a276ce] | (ID: 20180201171410)
==== Testing ====
* Manual - [https://wiki.mozilla.org/Releases/Firefox_59/Test_Plan/Beta/6 Firefox 59 Beta 6]
** 59.0b6-build1
*** {{mdone|}} Smoke and Exploratory Testing with focus on: Customization and Add-on Compatibility.
**** Detailed test results: [https://testrail.stage.mozaws.net/index.php?/reports/view/781 in TestRail].
*** {{mdone|}} [https://goo.gl/C8A1e4 Bug verification] with focus on the latest fixes to land in this build.
* Updates testing
** {{mdone|}} Updates on ''beta-cdntest'' channel , [https://public.etherpad-mozilla.org/p/beta-59.0b6-build1 report]
** {{mdone|}} Updates on ''beta'' channel , [https://public.etherpad-mozilla.org/p/beta-59.0b6-build1 report]
=== Beta 7 ===
==== Build Information ====
* Build 1: [https://archive.mozilla.org/pub/firefox/candidates/59.0b7-candidates/build1/ 59.0b7-build1] | [https://hg.mozilla.org/releases/mozilla-beta/rev/1a24837f9ed2 1a24837f9ed2] | (ID: 20180205211730)
==== Testing ====
* Manual - [https://wiki.mozilla.org/Releases/Firefox_59/Test_Plan/Beta/7 Firefox 59 Beta 7]
** 59.0b7-build1
*** {{mdone|}} Smoke and Exploratory Testing with focus on: Safe browsing and Graphics.
**** Detailed test results: [https://testrail.stage.mozaws.net/index.php?/plans/view/8180 in TestRail].
*** {{mdone|}} [https://goo.gl/tqqpnx Bug verification] with focus on the latest fixes to land in this build.
* Updates testing
** {{mdone|}} Updates on ''beta-cdntest'' channel , [https://public.etherpad-mozilla.org/p/beta-59.0b7-build1 report]
** {{mdone|}} Updates on ''beta'' channel , [https://public.etherpad-mozilla.org/p/beta-59.0b7-build1 report]
=== Beta 8 ===
==== Build Information ====
* Build 1: [https://archive.mozilla.org/pub/firefox/candidates/59.0b8-candidates/build2/ 59.0b8-build2] | [https://hg.mozilla.org/releases/mozilla-beta/rev/73ef186ad51a 73ef186ad51a] | (ID: 20180208193705)
==== Testing ====
* Manual - [https://wiki.mozilla.org/Releases/Firefox_59/Test_Plan/Beta/8 Firefox 59 Beta 8]
** 59.0b8-build2
*** {{mdone|}} Smoke and Exploratory Testing with focus on: Printing and Private Browsing.
**** Detailed test results: [https://testrail.stage.mozaws.net/index.php?/reports/view/788 in TestRail].
*** {{mdone|}} [https://goo.gl/tqqpnx Bug verification] with focus on the latest fixes to land in this build.
* Updates testing
** {{mdone|}} Updates on ''beta-cdntest'' channel , [https://public.etherpad-mozilla.org/p/beta-59.0b8-build2 report]
** {{mdone|}} Updates on ''beta'' channel , [https://public.etherpad-mozilla.org/p/beta-59.0b8-build2 report]
=== Beta 9 ===
==== Build Information ====
* Build 1: [https://archive.mozilla.org/pub/firefox/candidates/59.0b9-candidates/build1/ 59.0b9-build1] | [https://hg.mozilla.org/releases/mozilla-beta/rev/7dcfefd76e4c 7dcfefd76e4c] | (ID: 20180209162511)
==== Testing ====
* Manual - [https://wiki.mozilla.org/Releases/Firefox_59/Test_Plan/Beta/9 Firefox 59 Beta 9]
** 59.0b9-build1
*** {{mdone|}} Smoke and Exploratory Testing with focus on: Smart Card, Media playback & support and Toolbars & window controls.
**** Detailed test results: [https://testrail.stage.mozaws.net/index.php?/plans/view/8279 in TestRail].
*** {{mdone|}} [https://goo.gl/tqqpnx Bug verification] with focus on the latest fixes to land in this build.
* Updates testing
** {{mdone|}} Updates on ''beta-cdntest'' channel , [https://public.etherpad-mozilla.org/p/beta-59.0b9-build1 report]
** {{mdone|}} Updates on ''beta'' channel , [https://public.etherpad-mozilla.org/p/beta-59.0b9-build1 report]
=== Beta 10 ===
==== Build Information ====
* Build 3: [https://archive.mozilla.org/pub/firefox/candidates/59.0b10-candidates/build3/ 59.0b10-build3] | [https://hg.mozilla.org/releases/mozilla-beta/rev/b89f4d21c530 b89f4d21c530] | (ID: 20180215111455)
==== Testing ====
* Manual - [https://wiki.mozilla.org/Releases/Firefox_59/Test_Plan/Beta/10 Firefox 59 Beta 10]
** 59.0b10-build3
*** {{mdone|}} Smoke and Exploratory Testing with focus on: Migration, PDF viewer and Image compatibility & support.
**** Detailed test results: [https://testrail.stage.mozaws.net/index.php?/plans/view/8348 in TestRail].
*** {{mdone|}} [https://goo.gl/tqqpnx Bug verification] with focus on the latest fixes to land in this build.
* Updates testing
** {{mdone|}} Updates on ''beta-cdntest'' channel , [https://public.etherpad-mozilla.org/p/beta-59.0b10-build3 report]
** {{mdone|}} Updates on ''beta'' channel , [https://public.etherpad-mozilla.org/p/beta-59.0b10-build3 report]
=== Beta 11 ===
==== Build Information ====
* Build 3: [https://archive.mozilla.org/pub/firefox/candidates/59.0b11-candidates/build1/ 59.0b11-build1] | [https://hg.mozilla.org/releases/mozilla-beta/rev/daeb05a6586b daeb05a6586b] | (ID: 20180219114835)
==== Testing ====
* Manual - [https://wiki.mozilla.org/Releases/Firefox_59/Test_Plan/Beta/11 Firefox 59 Beta 11]
** 59.0b11-build1
*** {{mdone|}} Smoke and Exploratory Testing with focus on: Windows Child Mode, Download manager, Geolocation, Password Manager & APZ: Keyboard scrolling and Autoscrolling.
**** Detailed test results: [https://testrail.stage.mozaws.net/index.php?/reports/view/805 in TestRail].
*** {{mdone|}} [https://goo.gl/tqqpnx Bug verification] with focus on the latest fixes to land in this build.
* Updates testing
** {{mdone|}} Updates on ''beta-cdntest'' channel , [https://public.etherpad-mozilla.org/p/beta-59.0b11-build1 report]
** {{mdone|}} Updates on ''beta'' channel , [https://public.etherpad-mozilla.org/p/beta-59.0b11-build1 report]
=== Beta 12 ===
==== Build Information ====
* Build 1: [https://archive.mozilla.org/pub/firefox/candidates/59.0b12-candidates/build1/ 59.0b12-build1] | [https://hg.mozilla.org/releases/mozilla-beta/rev/abc5f0d35082 abc5f0d35082] | (ID: 20180222170353)
==== Testing ====
* Manual - [https://wiki.mozilla.org/Releases/Firefox_59/Test_Plan/Beta/12 Firefox 59 Beta 12]
** 59.0b12-build1
*** {{mdone|}} Smoke and Exploratory Testing with focus on: Plug-in compatibility, Downloads, Geolocation & Search.
**** Detailed test results: [https://testrail.stage.mozaws.net/index.php?/plans/view/8513 in TestRail].
*** {{mdone|}} [https://goo.gl/tqqpnx Bug verification] with focus on the latest fixes to land in this build.
* Updates testing
** {{mdone|}} Updates on ''beta-cdntest'' channel , [https://public.etherpad-mozilla.org/p/beta-59.0b12-build1 report]
** {{mdone|}} Updates on ''beta'' channel , [https://public.etherpad-mozilla.org/p/beta-59.0b12-build1 report]
=== Beta 13 ===
==== Build Information ====
* Build 1: [https://archive.mozilla.org/pub/firefox/candidates/59.0b13-candidates/build1/ 59.0b13-build1] | [https://hg.mozilla.org/releases/mozilla-beta/rev/db9096e8e9bd db9096e8e9bd] | (ID: 20180226180053)
==== Testing ====
* Manual - [https://wiki.mozilla.org/Releases/Firefox_59/Test_Plan/Beta/13 Firefox 59 Beta 13]
** 59.0b13-build1
*** {{mdone|}} Smoke and Exploratory Testing with focus on: Add-on compatibility & Activity Stream.
**** Detailed test results: [https://testrail.stage.mozaws.net/index.php?/plans/view/8546 in TestRail].
*** {{mdone|}} [https://goo.gl/tqqpnx Bug verification] with focus on the latest fixes to land in this build.
* Updates testing
** {{mdone|}} Updates on ''beta-cdntest'' channel , [https://public.etherpad-mozilla.org/p/beta-59.0b13-build1 report]
** {{mdone|}} Updates on ''beta'' channel , [https://public.etherpad-mozilla.org/p/beta-59.0b13-build1 report]
=== Beta 14 ===
==== Build Information ====
* Build 1: [https://archive.mozilla.org/pub/firefox/candidates/59.0b14-candidates/build1/ 59.0b14-build1] | [https://hg.mozilla.org/releases/mozilla-beta/rev/f83587c150ab f83587c150ab] | (ID: 20180301022608)
==== Testing ====
* Manual - [https://wiki.mozilla.org/Releases/Firefox_59/Test_Plan/Beta/14 Firefox 59 Beta 14]
** 59.0b14-build1
*** {{mdone|}} Smoke and Exploratory Testing with focus on: Crash reporting, Copy, paste, drag and drop, zoom indicator and dll interceptor.
**** Detailed test results: [https://testrail.stage.mozaws.net/index.php?/plans/view/8610 in TestRail].
*** {{mdone|}} [https://goo.gl/tqqpnx Bug verification] with focus on the latest fixes to land in this build.
* Updates testing
** {{mdone|}} Updates on ''beta-cdntest'' channel , [https://public.etherpad-mozilla.org/p/beta-59.0b14-build1 report]
** {{mdone|}} Updates on ''beta'' channel , [https://public.etherpad-mozilla.org/p/beta-59.0b14-build1 report]
=== 59.0 Release Candidate ===
==== Build Information ====
* Build 1: [https://archive.mozilla.org/pub/firefox/candidates/59.0b-candidates/build1/ 59.0-build1] | [https://hg.mozilla.org/releases/mozilla-beta/rev/2f6a4d2cf42c 2f6a4d2cf42c] | (ID: 20180305193205)
==== Testing ====
* Manual - [https://wiki.mozilla.org/Releases/Firefox_59/Test_Plan/Beta/59.0 Firefox 59 Release candidate]
** 59.0-build1
*** {{mdone|}} Smoke and Exploratory Testing with focus on: Web Compatibility.
**** Detailed test results: [https://testrail.stage.mozaws.net/index.php?/reports/view/822  in TestRail].
*** {{mdone|}} [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit#gid=468476081 Bug verification] with focus on the latest fixes to land in this build.
** {{mdone|}}  Updates on ''beta-cdntest'' channel [https://public.etherpad-mozilla.org/p/rc-59.0-build1 report]
** {{mdone|}} Updates on ''beta'' channel [https://public.etherpad-mozilla.org/p/rc-59.0-build1 report]
** {{mnotstarted|}} Updates on ''release-cdntest'' channel [https://public.etherpad-mozilla.org/p/rc-59.0-build1 report]
** {{mnotstarted|}} Updates on ''release'' channel [https://public.etherpad-mozilla.org/p/rc-59.0-build1 report]
=== 59.0 Release Candidate 2 ===
==== Build Information ====
* Build 4: [https://archive.mozilla.org/pub/firefox/candidates/59.0b-candidates/build1/ 59.0-build4] | [https://hg.mozilla.org/releases/mozilla-beta/rev/e9128973a480 e9128973a480] | (ID: 20180307211018)
==== Testing ====
* Manual - [https://wiki.mozilla.org/Releases/Firefox_59/Test_Plan/Beta/59.0RC2 Firefox 59 Release candidate 2]
** 59.0-build4
*** {{mdone|}} Smoke and Exploratory Testing with focus on: Web Compatibility.
**** Detailed test results: [https://testrail.stage.mozaws.net/index.php?/reports/view/826  in TestRail].
*** {{mdone|}} [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit#gid=468476081 Bug verification] with focus on the latest fixes to land in this build.
** {{mdone|}}  Updates on ''beta-cdntest'' channel [https://public.etherpad-mozilla.org/p/rc-59.0-build4 report]
** {{mnotstarted|}} Updates on ''beta'' channel [https://public.etherpad-mozilla.org/p/rc-59.0-build4 report]
** {{mnotstarted|}} Updates on ''release-cdntest'' channel [https://public.etherpad-mozilla.org/p/rc-59.0-build4 report]
** {{mnotstarted|}} Updates on ''release'' channel [https://public.etherpad-mozilla.org/p/rc-59.0-build4 report]
=== 59.0 Release Candidate 3===
==== Build Information ====
* Build 5: [https://archive.mozilla.org/pub/firefox/candidates/59.0b-candidates/build1/ 59.0-build5] | [https://hg.mozilla.org/releases/mozilla-beta/rev/c61f5f5ead48 c61f5f5ead48] | (ID: 20180310025718)
==== Testing ====
* Manual - [https://docs.google.com/document/d/1q9Gng6Ul9VKEoVcWeER_25Dwd7VBcgySHH0lgmGoWL8/edit# Firefox 59 Release candidate 3]
** 59.0-build5
*** {{mdone|}} Smoke and Exploratory Testing with focus on: Search suggestions.
**** Detailed test results: [https://docs.google.com/document/d/1y-RuZeitBIm1R2dhDM_3SCglc3ShhhQUTGDMGJ5gKPs/edit#heading=h.c1k9kvic8ruu in gdoc].
*** {{mdone|}} [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit#gid=468476081 Bug verification] with focus on the latest fixes to land in this build.
** {{mdone|}}  Updates on ''beta-cdntest'' channel [https://public.etherpad-mozilla.org/p/rc-59.0-build5 report]
** {{mdone|}} Updates on ''beta'' channel [https://public.etherpad-mozilla.org/p/rc-59.0-build5 report]
** {{mdone|}} Updates on ''release-cdntest'' channel [https://public.etherpad-mozilla.org/p/rc-59.0-build5 report]
** {{mdone|}} Updates on ''release'' channel [https://public.etherpad-mozilla.org/p/rc-59.0-build5 report]
=== 59.0.1===
==== Build Information ====
* Build 1: [https://archive.mozilla.org/pub/firefox/candidates/59.0.1-candidates/build1/ 59.0.1-build1] | [https://hg.mozilla.org/releases/mozilla-beta/rev/3db9e3d52b17 3db9e3d52b17] | (ID: 20180315233128)
==== Testing ====
* Manual - [https://wiki.mozilla.org/Releases/Firefox_59/Test_Plan/Beta/59.0.1 Firefox 59.0.1]
** 59.0.1-build1
*** {{mdone|}} Smoke and Exploratory Testing with focus on: Web compatibility.
**** Detailed test results: [https://testrail.stage.mozaws.net/index.php?/plans/view/8857 in TestRail].
*** {{mdone|}} [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit#gid=468476081 Bug verification] with focus on the latest fixes to land in this build.
** {{mdone|}} Updates on ''release-localtest'' channel [https://public.etherpad-mozilla.org/p/Firefox_59.0.1-build1 report]
** {{mdone|}} Updates on ''release-cdntest'' channel [https://public.etherpad-mozilla.org/p/Firefox_59.0.1-build1 report] (including WNP)
** {{mdone|}} Updates on ''release'' channel [https://public.etherpad-mozilla.org/p/Firefox_59.0.1-build1 report] (including WNP)
=== 59.0.2===
==== Build Information ====
* Build 1: [https://archive.mozilla.org/pub/firefox/candidates/59.0.2-candidates/build1/ 59.0.2-build1] | [https://hg.mozilla.org/releases/mozilla-release/rev/239e434d6d2b 239e434d6d2b] | (ID: 20180323154952)
==== Testing ====
* Manual - [https://wiki.mozilla.org/Releases/Firefox_59/Test_Plan/Beta/59.0.2 Firefox 59.0.2]
** 59.0.2-build1
*** {{mdone|}} Smoke and Exploratory Testing with focus on: Web compatibility.
**** Detailed test results: [https://testrail.stage.mozaws.net/index.php?/plans/view/8857 in TestRail].
*** {{mdone|}} [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit#gid=468476081 Bug verification] with focus on the latest fixes to land in this build.
** {{mdone|}} Updates on ''release-localtest'' channel [https://public.etherpad-mozilla.org/p/Firefox_59.0.2-build1 report] (including WNP)
** {{mdone|}} Updates on ''release-cdntest'' channel [https://public.etherpad-mozilla.org/p/Firefox_59.0.2-build1 report] (including WNP)
** {{mnotstarted|}} Updates on ''release'' channel [https://public.etherpad-mozilla.org/p/Firefox_59.0.2-build1 report] (including WNP)
=== 59.0.3===
==== Build Information ====
* Build 1: [https://archive.mozilla.org/pub/firefox/candidates/59.0.3-candidates/build1/ 59.0.3-build1] | [https://hg.mozilla.org/releases/mozilla-release/rev/6b51784853e4 6b51784853e4] | (ID: 20180427210249)
==== Testing ====
* Manual - [https://wiki.mozilla.org/Releases/Firefox_59/Test_Plan/Beta/59.0.3 Firefox 59.0.3]
** 59.0.3-build1
*** {{mdone|}} Smoke and Exploratory Testing with focus on: Web compatibility.
**** Detailed test results: [https://testrail.stage.mozaws.net/index.php?/plans/view/results in TestRail].
*** {{mdone|}} [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit#gid=468476081 Bug verification] with focus on the latest fixes to land in this build.
** {{mnotstarted|}} Updates on ''release-localtest'' channel [https://public.etherpad-mozilla.org/p/Firefox_59.0.3-build1 report] (agreed with jcristau to skip this)
** {{mdone|}} Updates on ''release-cdntest'' channel [https://public.etherpad-mozilla.org/p/Firefox_59.0.3-build1 report] (including WNP)
** {{mnotstarted|}} Updates on ''release'' channel [https://public.etherpad-mozilla.org/p/Firefox_59.0.3-build1 report] (including WNP)

Latest revision as of 13:55, 30 April 2018

Summary

This page is to track testing of Firefox 59 from mozilla-central (Nightly) through to mozilla-release. This is intended to be a minimal set of information to help Firefox QA understand and communicate the quality of upcoming releases. Our current walkthrough of the release and QA process might be changing due to recent reorgs.

The QA efforts are mostly split up between two teams:

  • Engineering QA
    • team reports to Ryan VanderMeulen and Tom Grabowski
    • the Project Manager (and main contact) for the team is Rares Bologa
    • team is based in Romania (most part) and Las Vegas (5 people)
    • the team mainly tests the Desktop app, but covers the Mobile (Android, iOS) side as well
    • main responsibilities:
      • Feature Testing - new Firefox features, prior to landing and during Nightly
      • Bug Triage - older as well as incoming Unconfirmed bugs
      • Bug Investigation - various investigation as requested by other people (e.g. finding steps, setting component, finding regression windows)
      • Misc - other miscellaneous requests
  • Release QA Desktop and Release QA Mobile
    • team reports to Ryan VanderMeulen and Tom Grabowski
    • the Project Manager (and main contact) for the team is Andrei Vaida
    • team is based in Romania
    • main responsibilities:
      • Build Validation - mostly Beta, Release, and ESR builds (including automated ondemand update, and throttling tests)
      • Feature Testing - new features not covered by the Engineering QA team, and those covered by the Engineering QA team after they enter Beta
      • Bug Verification - verifying the most important / high risk fixes, with focus on those closer to release
      • Bug Investigation - various investigation as requested by other people (e.g. finding steps, setting component, finding regression windows)
      • Misc - other miscellaneous requests

Schedule

Error in widget Widget:Google Calendar: Unable to load template 'wiki:Widget:Google Calendar'
Milestone Date Checks
Nightly - Conduct daily bug triage, regular build validation, and feature testing
Beta - Feature testing and sign off every beta release, regular testing of DevEdition builds
Release - Review tests and sign off release

Please consult the Rapid Release Calendar for more information.

Meetings

Meeting Purpose When Vidyo Room IRC Backchannel Notes
Channel Status Raise quality concerns with the Release Management team 10:00 AM Pacific on Tuesday & 08:00 AM Pacific on Thursday ReleaseCoordination #planning Notes archive

Nightly

While Firefox 59 is on the Nightly channel, QA will mainly focus on:

  • Feature Testing - EngQA team (for Firefox features assigned to them) + RelQA team (for other new features - Platform or Firefox)
  • Regular Nightly validation - 3 rounds per cycle - EngQA team + RelQA team, alternately
  • Triaging newly reported bugs - EngQA team
  • Bug Verification - both teams - with main focus on: fixes related to new features, fixes tracked as Sprint work, fixes for major bugs

Reference: https://wiki.mozilla.org/QA/Desktop_Firefox/Walkthroughs/Release_Coordination#Nightly

End of cycle status
  • Total bugs filed affecting 59: 3362
  • Bugs fixed / verified on 59: 2453 / 190
  • Conditional signoff for merge: <date>, <person> (email sent to release-drivers at
  • QE signoff: N/A

Beta

During the Beta cycle, there are several beta releases. Communication with the release coordination, release engineering, and automation teams is key.

While Firefox 59 is on the Beta channel, QA will mainly focus on:

  • Signing off each Beta build prior to being pushed to the beta channel - RelQA
  • Feature Testing - RelQA team
    • Focus is on fixes that land in Beta, and Regression and Smoke testing
  • Regular validation of DevEdition builds - about once every two weeks
  • Triaging newly reported bugs - EngQA team
  • Bug Verification - mostly RelQA - with main focus on: fixes related to new features, fixes for major bugs, fixes assessed as high risk

QA also analyzes the results of OnDemand Update tests running on Jenkins. We create config files for the update tests, check their validity, and run them on at least two channels for each beta release cycle.

Pivotal bugs may need verification during each beta cycle.

  • Link to release notes. review & update.
  • Crash evaluation/escalation.
  • Feature evaluation.

QA Activities

QA wanted

The following tracks bugs which have requests for urgent QA assistance.

  • Choose a bug to test from the list below
  • Read it and make sure you understand what information is needed (this may be steps or URLs to reproduce, a regression range, or something else altogether)
  • Use the latest build of this Firefox version and test until you've discovered the information necessary. Ask for help if you need it.
  • Report this information to the bug and remove the qawanted keyword

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Stability

This section tracks bugs representing the most urgent stability issues. Consult this wiki for more information.

  • Ensure new and explosive crashes have actionable bug reports which are in progress
  • Top Crashes - ensure the highest ranked crashes have actionable bug reports which are in progress
  • Untriaged Bugs - ensure bugs are actionable and in the correct component
  • topcrash bugs

Unconfirmed Bug Triage

The following tracks triage and testing of incoming, unconfirmed bug reports.

  • Choose a bug to test from the list below
  • Use the latest Nightly build to confirm the bug reproduces
  • Flag with reporter with needinfo if you cannot reproduce and need more information
  • Set the status to NEW and ensure the bug is in the appropriate component if you can reproduce the bug (provide as much detail about your testing as possible)
  • If it's a Firefox bug you may want to add the firefox-backlog=? flag to it.

Queries:

Verification Needed Bugs

  • Review the bugs below in the Pending Triage section and select a bug you want to test
  • Make sure you understand the bug before you begin testing it
  • Test on a previously known broken build to make sure you can reproduce the bug
  • Test on the latest build from the branches which are fixed to confirm the bug no longer reproduces
  • Mark the bug VERIFIED FIXED if the bug appears to be fixed and set the appropriate status-firefox flag to verified
  • Reopen the bug if it still reproduces

Iteration Development

Firefox development during Nightly works on two-week iterations as of 2014. This section tracks the bugs selected for development in these iterations. Please consult the Iteration Dev Walkthrough and the Firefox/IterativeDevelopment page for more information on this process.

  • Ensure all bugs are marked qe-verify:+ or -
  • Ensure all qe-verify:+ bugs have a QA Contact (The Feature Owner page may help here)
  • Ensure all selected bugs are VERIFIED before the end of the iteration
  • Anything incomplete at the end of the iteration will be automatically carried over to the next iteration

Queries:

Feature status

Milestones

Regular Nightly Validation

SIGN-OFF DATE SIGN-OFF STATUS REASON(S) MAJOR ISSUES MINOR ISSUES TEST REPORT
2017-11-23 GREEN
  • 1 new, low-impact bug uncovered: bug 1419819.
  • Known (old) bugs are low-impact.
 --- Bugzilla link TestRail link
2017-12-15 GREEN
  • 3 new bugs uncovered, all of them low-impact: Bugzilla link.
  • Known (old) bugs are low-impact.
 --- Bugzilla link TestRail link
2018-01-11 TBD tbd tbd tbd tbd

Nightly Migration

Pre-sign-off Checks:

  • [NOT STARTED] Features - all scoped features are signed off as ready for Beta by owners. (If you haven't heard anything to the contrary, that is probably true, but make sure we don't know of blocking issues.)

Post-sign-off Checks (as soon as updates are enabled again):

  • None yet for migration of Nightly to Beta

Developer Edition

Note: Developer Edition builds are usually signed off once every two weeks.
BUILD SIGN OFF ENCOUNTERED BUGS
Status Reason(s) Test Report New Known (old)
59.0b1-build11 GREEN
  • No new issues uncovered.
  • Known issues are low-impact.
 TestRail link
  • Regressions: none.
  • Non-regressions: none.
  • High-impact: none.
  • Low-impact: none.
59.0b2-build1 GREEN
  • One new issue uncovered: 1431674, recent regression, not a blocker.
  • Known issues are low-impact.
 TestRail link
  • Regressions: 1431674.
  • Non-regressions: none.
  • High-impact: none.
  • Low-impact: link.
59.0b5-build1 GREEN
  • No new issues uncovered during testing
  • Known issues are low-impact.
 TestRail link
  • Regressions: none.
  • Non-regressions: none.
  • High-impact: none.
  • Low-impact: link.
59.0b9-build1 GREEN
  • Regressions: none.
  • Non-regressions: none.
 TestRail link
  • Regressions: none.
  • Non-regressions: none.
  • High-impact: none.
  • Low-impact: list.
59.0b13-build1 GREEN
  • One new issue uncovered: 1441843, recent regression, not a blocker.
  • Known issues are low-impact.
 TestRail link
  • Regressions: none.
  • Non-regressions: none.
  • High-impact: none.
  • Low-impact: list.

Beta

Note: Beta builds are usually signed off twice a week.

Beta 3

Build Information

Testing

  • Manual - Firefox 59 Beta 3
    • 59.0b3-build2
      • [DONE] Smoke and Exploratory Testing with focus on: Form Autofill V2, PDFium printing on Windows and Mac OS X signing.
      • [DONE] Bug verification with focus on the latest fixes to land in this build.
  • Updates testing
    • [DONE] Updates on beta-cdntest channel , report
    • [DONE] Updates on beta channel , report

Beta 4

Build Information

Testing

  • Manual - Firefox 59 Beta 4
    • 59.0b4-build1
      • [DONE] Smoke and Exploratory Testing with focus on: Graphics and Search.
      • [DONE] Bug verification with focus on the latest fixes to land in this build.
  • Updates testing
    • [DONE] Updates on beta-cdntest channel , report
    • [DONE] Updates on beta channel , report

Beta 5

Build Information

Testing

  • Manual - Firefox 59 Beta 5
    • 59.0b5-build1
      • [DONE] Smoke and Exploratory Testing with focus on: Web compatibility and Bookmarks.
      • [DONE] Bug verification with focus on the latest fixes to land in this build.
  • Updates testing
    • [DONE] Updates on beta-cdntest channel , report
    • [DONE] Updates on beta channel , report

Beta 6

Build Information

Testing

  • Manual - Firefox 59 Beta 6
    • 59.0b6-build1
      • [DONE] Smoke and Exploratory Testing with focus on: Customization and Add-on Compatibility.
      • [DONE] Bug verification with focus on the latest fixes to land in this build.
  • Updates testing
    • [DONE] Updates on beta-cdntest channel , report
    • [DONE] Updates on beta channel , report

Beta 7

Build Information

Testing

  • Manual - Firefox 59 Beta 7
    • 59.0b7-build1
      • [DONE] Smoke and Exploratory Testing with focus on: Safe browsing and Graphics.
      • [DONE] Bug verification with focus on the latest fixes to land in this build.
  • Updates testing
    • [DONE] Updates on beta-cdntest channel , report
    • [DONE] Updates on beta channel , report

Beta 8

Build Information

Testing

  • Manual - Firefox 59 Beta 8
    • 59.0b8-build2
      • [DONE] Smoke and Exploratory Testing with focus on: Printing and Private Browsing.
      • [DONE] Bug verification with focus on the latest fixes to land in this build.
  • Updates testing
    • [DONE] Updates on beta-cdntest channel , report
    • [DONE] Updates on beta channel , report

Beta 9

Build Information

Testing

  • Manual - Firefox 59 Beta 9
    • 59.0b9-build1
      • [DONE] Smoke and Exploratory Testing with focus on: Smart Card, Media playback & support and Toolbars & window controls.
      • [DONE] Bug verification with focus on the latest fixes to land in this build.
  • Updates testing
    • [DONE] Updates on beta-cdntest channel , report
    • [DONE] Updates on beta channel , report

Beta 10

Build Information

Testing

  • Manual - Firefox 59 Beta 10
    • 59.0b10-build3
      • [DONE] Smoke and Exploratory Testing with focus on: Migration, PDF viewer and Image compatibility & support.
      • [DONE] Bug verification with focus on the latest fixes to land in this build.
  • Updates testing
    • [DONE] Updates on beta-cdntest channel , report
    • [DONE] Updates on beta channel , report

Beta 11

Build Information

Testing

  • Manual - Firefox 59 Beta 11
    • 59.0b11-build1
      • [DONE] Smoke and Exploratory Testing with focus on: Windows Child Mode, Download manager, Geolocation, Password Manager & APZ: Keyboard scrolling and Autoscrolling.
      • [DONE] Bug verification with focus on the latest fixes to land in this build.
  • Updates testing
    • [DONE] Updates on beta-cdntest channel , report
    • [DONE] Updates on beta channel , report

Beta 12

Build Information

Testing

  • Manual - Firefox 59 Beta 12
    • 59.0b12-build1
      • [DONE] Smoke and Exploratory Testing with focus on: Plug-in compatibility, Downloads, Geolocation & Search.
      • [DONE] Bug verification with focus on the latest fixes to land in this build.
  • Updates testing
    • [DONE] Updates on beta-cdntest channel , report
    • [DONE] Updates on beta channel , report

Beta 13

Build Information

Testing

  • Manual - Firefox 59 Beta 13
    • 59.0b13-build1
      • [DONE] Smoke and Exploratory Testing with focus on: Add-on compatibility & Activity Stream.
      • [DONE] Bug verification with focus on the latest fixes to land in this build.
  • Updates testing
    • [DONE] Updates on beta-cdntest channel , report
    • [DONE] Updates on beta channel , report

Beta 14

Build Information

Testing

  • Manual - Firefox 59 Beta 14
    • 59.0b14-build1
      • [DONE] Smoke and Exploratory Testing with focus on: Crash reporting, Copy, paste, drag and drop, zoom indicator and dll interceptor.
      • [DONE] Bug verification with focus on the latest fixes to land in this build.
  • Updates testing
    • [DONE] Updates on beta-cdntest channel , report
    • [DONE] Updates on beta channel , report

59.0 Release Candidate

Build Information

Testing

  • Manual - Firefox 59 Release candidate
    • 59.0-build1
      • [DONE] Smoke and Exploratory Testing with focus on: Web Compatibility.
      • [DONE] Bug verification with focus on the latest fixes to land in this build.
    • [DONE] Updates on beta-cdntest channel report
    • [DONE] Updates on beta channel report
    • [NOT STARTED] Updates on release-cdntest channel report
    • [NOT STARTED] Updates on release channel report

59.0 Release Candidate 2

Build Information

Testing

  • Manual - Firefox 59 Release candidate 2
    • 59.0-build4
      • [DONE] Smoke and Exploratory Testing with focus on: Web Compatibility.
      • [DONE] Bug verification with focus on the latest fixes to land in this build.
    • [DONE] Updates on beta-cdntest channel report
    • [NOT STARTED] Updates on beta channel report
    • [NOT STARTED] Updates on release-cdntest channel report
    • [NOT STARTED] Updates on release channel report

59.0 Release Candidate 3

Build Information

Testing

  • Manual - Firefox 59 Release candidate 3
    • 59.0-build5
      • [DONE] Smoke and Exploratory Testing with focus on: Search suggestions.
      • [DONE] Bug verification with focus on the latest fixes to land in this build.
    • [DONE] Updates on beta-cdntest channel report
    • [DONE] Updates on beta channel report
    • [DONE] Updates on release-cdntest channel report
    • [DONE] Updates on release channel report

59.0.1

Build Information

Testing

  • Manual - Firefox 59.0.1
    • 59.0.1-build1
      • [DONE] Smoke and Exploratory Testing with focus on: Web compatibility.
      • [DONE] Bug verification with focus on the latest fixes to land in this build.
    • [DONE] Updates on release-localtest channel report
    • [DONE] Updates on release-cdntest channel report (including WNP)
    • [DONE] Updates on release channel report (including WNP)

59.0.2

Build Information

Testing

  • Manual - Firefox 59.0.2
    • 59.0.2-build1
      • [DONE] Smoke and Exploratory Testing with focus on: Web compatibility.
      • [DONE] Bug verification with focus on the latest fixes to land in this build.
    • [DONE] Updates on release-localtest channel report (including WNP)
    • [DONE] Updates on release-cdntest channel report (including WNP)
    • [NOT STARTED] Updates on release channel report (including WNP)

59.0.3

Build Information

Testing

  • Manual - Firefox 59.0.3
    • 59.0.3-build1
      • [DONE] Smoke and Exploratory Testing with focus on: Web compatibility.
      • [DONE] Bug verification with focus on the latest fixes to land in this build.
    • [NOT STARTED] Updates on release-localtest channel report (agreed with jcristau to skip this)
    • [DONE] Updates on release-cdntest channel report (including WNP)
    • [NOT STARTED] Updates on release channel report (including WNP)
Retrieved from "https://wiki.allizom.org/index.php?title=Releases/Firefox_59/Test_Plan&oldid=1193012"