Releases/Firefox 72/Test Plan: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(correction of summary)
 
(9 intermediate revisions by 3 users not shown)
Line 85: Line 85:
== QA activities ==
== QA activities ==
=== New feature readiness ===
=== New feature readiness ===
All new features tested by QA and shipping with Firefox 71 are available [https://docs.google.com/spreadsheets/u/1/d/1aRRjQiFo4jT3fr1dJ-ZDsYa8SrQ9PzYbUCojataMFls/edit#gid=0 here].
All new features tested by QA and shipping with Firefox 72 are available [https://docs.google.com/spreadsheets/d/1px85HjaQHgpfq68bdMBbhusIsEZXmP_Ig3_KlAoyFRc/edit#gid=0 here].


The feature testing process followed by QA is available [https://wiki.mozilla.org/QA/Feature_Testing_v2 here].
The feature testing process followed by QA is available [https://wiki.mozilla.org/QA/Feature_Testing_v2 here].
Line 308: Line 308:
<div class="mw-collapsible-content">
<div class="mw-collapsible-content">
* Build: 72.0-build3 (https://hg.mozilla.org/releases/mozilla-release/rev/d88c5f2e59dccf2bfd345e66e548f2f245561a00  d88c5f2e59dccf2bfd345e66e548f2f245561a00], 20200102182448) - <font style="color:green; font-weight:bold;">GREEN</font>
* Build: 72.0-build3 (https://hg.mozilla.org/releases/mozilla-release/rev/d88c5f2e59dccf2bfd345e66e548f2f245561a00  d88c5f2e59dccf2bfd345e66e548f2f245561a00], 20200102182448) - <font style="color:green; font-weight:bold;">GREEN</font>
*** {{notstarted|}} Smoke, regression, exploratory → [https://testrail.stage.mozaws.net/index.php?/plans/view/23975 TestRail]
*** {{mdone|}} Smoke, regression, exploratory → [https://testrail.stage.mozaws.net/index.php?/plans/view/23975 TestRail]
*** {{notstarted|}} Smoke testing on Ubuntu snap package → Installation
*** {{mdone|}} Smoke testing on Ubuntu snap package → Installation
*** {{inprogress|}} Bug fix verification – [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit#gid=914556339 Bug list]
*** {{mdone|}} Bug fix verification – [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit#gid=914556339 Bug list]
** '''Update tests (manual)''':
** '''Update tests (manual)''':
*** {{notstarted|}} release-cdntest, beta-cdntest → [https://drive.google.com/open?id=1750v087PYtfxaiev26WChcNnUYGHpUl4NWu9gIn786A report]
*** {{notstarted|}} release-cdntest, beta-cdntest → [https://drive.google.com/open?id=1750v087PYtfxaiev26WChcNnUYGHpUl4NWu9gIn786A report]
*** {{notstarted|}}  release → [https://drive.google.com/open?id=1750v087PYtfxaiev26WChcNnUYGHpUl4NWu9gIn786A report]
*** {{mdone|}}  release → [https://drive.google.com/open?id=1750v087PYtfxaiev26WChcNnUYGHpUl4NWu9gIn786A report]
</div></div>
 
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">72.0.1</div>
<div class="mw-collapsible-content">
* Build: 72.0.1-build1 ([https://hg.mozilla.org/releases/mozilla-release/rev/8260da04c9b1 8260da04c9b1], 20200107212822) - <font style="color:green; font-weight:bold;">GREEN</font>
*** {{mdone|}} Smoke, regression, exploratory → [https://testrail.stage.mozaws.net/index.php?/plans/view/24161 TestRail]
*** {{mdone|}} Smoke testing on Ubuntu snap package → Installation
*** {{mdone|}} Bug fix verification – [https://bugzilla.mozilla.org/show_bug.cgi?id=1607443 Chempsill bug]
** '''Update tests (manual)''':
*** {{mdone|}} release-cdntest → [https://docs.google.com/spreadsheets/u/1/d/1U4qR2J3wh-ntiFZjtb-Fvct54Q2jO6M4Suj-NK0uoss/edit?usp=drive_web&ouid=1025689521899394819 report]
*** {{mdone|}} release → [https://docs.google.com/spreadsheets/u/1/d/1U4qR2J3wh-ntiFZjtb-Fvct54Q2jO6M4Suj-NK0uoss/edit?usp=drive_web&ouid=1025689521899394819 report]
</div></div>
 
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto; overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">72.0.2</div>
<div class="mw-collapsible-content">
* Build: 72.0.2-build1 ([https://hg.mozilla.org/releases/mozilla-release/rev/25e0edbb0a613c3bf794c93ba3aa0985d29d5ef4 25e0edbb0a613c3bf794c93ba3aa0985d29d5ef4], 20200117190643) - <font style="color:green; font-weight:bold;">GREEN</font>
*** {{mdone|}} Smoke, regression, exploratory → [https://testrail.stage.mozaws.net/index.php?/plans/view/24655 TestRail]
*** {{mdone|}} Smoke testing on Ubuntu snap package → Installation
*** {{mdone|}} Bug fix verification – [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=1601905%201602726%201604249%201604490%201607494%201608478&list_id=15077106 Driver bugs]
** '''Update tests (manual)''':
*** {{mdone|}} release-cdntest → [https://docs.google.com/spreadsheets/d/1b-z9PvqohA47-DiUUi9fZbDv8B_nxk2hUlyDjfhyqlw/edit#gid=850719639 report]
*** {{mdone|}} release → [https://docs.google.com/spreadsheets/d/1b-z9PvqohA47-DiUUi9fZbDv8B_nxk2hUlyDjfhyqlw/edit#gid=850719639 report]
</div></div>
</div></div>

Latest revision as of 07:16, 21 January 2020

Summary

The purpose of this page is to track the testing of Firefox 72 from Nightly to Release, and post Release if needed.

Release schedule

Status meetings

The Channel Status meeting is used to raise quality concerns with the Release Management team.

QA points of contact

Communication channels

Firefox for Desktop

Engineering QA
  • Reporting to: Tom Grabowski, Tania Maity
  • Softvision Manager: Rares Bologa
  • Team Leads: Brindusa Tot
  • Offices: Romania (EEST), Argentina (GMT-3)
  • Responsible for: new feature testing, Nightly build readiness, Nightly build regression testing, bug triage, bug verification, bug investigation
Release QA
  • Reporting to: Tom Grabowski, Tania Maity
  • Softvision Manager: Andrei Vaida
  • Team Leads: Bogdan Maris, Cornel Ionce
  • Office: Romania (EEST)
  • Responsible for: new feature testing, Beta build readiness, Developer Edition build readiness, ESR build readiness, RC build readiness, Dot release build readiness, Chemspill build readiness, bug verification, bug investigation

Firefox for Mobile

Release QA
Firefox for iOS QA

AMO, WebExtensions

Add-ons QA
  • Reporting to: Krupa Raj
  • Softvision Manager: Rares Bologa
  • Team Leads: Victor Carciu, Madalin Cotetiu
  • Office: Romania (EEST)
  • Responsible for: AMO feature testing, AMO weekly release, AMO Sanity, AMO bug verification, Webextensions features, Webextensions bug triage and verification, Telemetry

Experiments, Normandy, Subscription services, and other Firefox features

Experiments QA
  • Reporting to: Krupa Raj
  • Softvision Manager: Rares Bologa
  • Team Leads: Paul Oiegas
  • Office: Romania (EEST)
  • Responsible for: testing Experiments, Normandy, Experimenter website, Activity Stream, Screenshots, Send (Web & Android) and Lockbox Desktop.

QA activities

New feature readiness

All new features tested by QA and shipping with Firefox 72 are available here.

The feature testing process followed by QA is available here.

Bug fix verification

  • Daily verification of uplifted bugs - these are uplifts, triaged daily and verified under ~48hrs
  • Ongoing verification of Nightly bugs - this is the general bug verification done throughout the Nightly cycle; it sometimes overlaps with uplifts verification; also involves active monitoring of qe-verify? bugs
  • Ongoing verification of Beta bugs - this is the general bug verification done throughout the Beta cycle; it sometimes overlaps with uplifts verification; also involves active monitoring of qe-verify? bugs
  • On-demand verification of bugs - these are special requests (ideally coming in through PI, but also through Bugzilla needinfo?, chat or emails)
  • Periodical verification of security bugs - these are post-critsmash bugs, triaged and verified ~every 2wks

Unconfirmed bug triage

This is the triage conducted by Engineering QA.

Security bug triage

This is the post-CritSmash triage conducted by Engineering QA (on Nightly) and Release QA (on Beta, Release, ESR).

Bug investigation

This is the daily check and investigation of new regressionwindow-wanted bugs, conducted by Engineering QA and Release QA.

Nightly 72 milestones

[early/mid cycle] Nightly build readiness

Scope: assess build readiness for shipping to Beta by testing Nightly periodically and providing a list of issues (if any) that need to be addressed before the merge (more on this).

Nightly 72.0a1 (2019-05-30)
  • Build: 72.0a1 (commit, ID) - NOT STARTED
    • [DONE] Functional testing (manual) → report
      • New bugs
        • Regressions: --
        • Non-regressions: --
      • Known bugs
        • High-impact: --
        • Low/medium-impact: [link bug list]
Nightly 72.0a1 (2019-06-13)
  • Build: 72.0a1 (commit, ID) - GREEN
    • [DONE] Functional testing (manual) → report
      • New bugs
        • Regressions: --
        • Non-regressions: --
      • Known bugs
        • High-impact: --
        • Low/medium-impact: [link bug list]

[prior to merge] Nightly regression testing

Nightly 69.0a1 (date)
  • Build: 73.0a1 (commit, ID) - Sign off_Status
    • [NOT STARTED] Regression testing (manual) → [report]
      • New bugs
        • Regressions: ID1, ID2, IDn
        • Non-regressions: ID1, ID2, IDn
      • Known bugs
        • High-impact: ID1, ID2, IDn
        • Low/medium-impact: [bug list]

[optional] [prior to merge] Nightly new feature smoke testing

Note: this is optional, based on available QA bandwidth.

Beta 72 milestones

Beta/DevEdition build readiness

Developer Edition build readiness

DevEd 72.0b4
  • Build: 72.0b4-build1 (8919727c131ee5a5389d28f2f951b6c3ef9953e0, 20191206183317) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Build Branding , Build Configuration , User Profile , Build interface , Build stability and functionality , Sync , Default Browser, Webextensions → TestRail

Beta build readiness

Beta 72.0b1
  • Build: 72.0b1-build1 (fbd5649f1f88, 20191202142314) - GREEN
    • Functional tests (manual):
    • Update tests (manual):
Beta 72.0b2
  • Build: 72.0b2-build1 (1157109d1975, 20191203141951) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke testing on Address Bar and Search, Audio/Video, Graphics, Crash Reporting and Plug-ins → TestRail
      • [DONE] Bug fix verification
    • Update tests (manual):
Beta 72.0b3
  • Build: 72.0b3-build1 (229d39ef2711, 20191205201734) - GREEN
    • Functional tests (manual):
    • Update tests (manual):
      • [DONE] beta, aurora → report
Beta 72.0b4
  • Build: 72.0b4-build1 (229d39ef2711, 20191206183317) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke testing on Security and Privacy ,Drag and Drop, Scrolling, Panning and Zooming and Geolocation → TestRail
      • [IN PROGRESS] Bug fix verification
    • Update tests (manual):
      • [DONE] beta, aurora → report
Beta 72.0b5
  • Build: 72.0b5-build1 (ebe06d4050d9, 20191210230245) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke testing on Security and Privacy ,Theme and Toolbar Customization, PDF Viewer and Images → TestRail
      • [IN PROGRESS] Bug fix verification
    • Update tests (manual):
      • [IN PROGRESS] beta, aurora → report
Beta 72.0b6
  • Build: 72.0b6-build1 (baa895188e29, 20191212210843) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke testing on Printing ,Security and Privacy, Browser Window , Autocomplete & Autofill and Tab Bar options→ TestRail
      • [DONE] Bug fix verification
    • Update tests (manual):
      • [DONE] beta, aurora → report
Beta 72.0b7
  • Build: 72.0b7-build1 (cd6d215cb8fa, 20191213132525) - GREEN
    • Functional tests (manual):
    • Update tests (manual):
      • [DONE] beta, aurora → report
Beta 72.0b8
  • Build: 72.0b8-build1 ([1], 20191218033533) - GREEN
    • Functional tests (manual):
    • Update tests (manual):
      • [DONE] beta, aurora → report
Beta 72.0b9
  • Build: 72.0b9-build1 (a15c8d0d512e, 20191219234059) - GREEN
    • Functional tests (manual):
    • Update tests (manual):
      • [DONE] beta, aurora → report
Beta 72.0b10
  • Build: 72.0b10-build1 (0941e84243b2, 20191221003008) - GREEN
    • Functional tests (manual):
    • Update tests (manual):
      • [DONE] beta, aurora → report
Beta 72.0b11
  • Build: 72.0b11-build1 (d5822b0bc63d, 20191227034945) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke testing on Tabbed Browser, Password Manager , Preferences , Menus and Firefox Screenshots → TestRail
      • [DONE] Bug fix verification
    • Update tests (manual):
      • [DONE] beta, aurora → report

Release 72 build readiness

RC1 72.0
RC2 72.0
72.0.1
  • Build: 72.0.1-build1 (8260da04c9b1, 20200107212822) - GREEN
      • [DONE] Smoke, regression, exploratory → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → Installation
      • [DONE] Bug fix verification – Chempsill bug
    • Update tests (manual):
72.0.2
Retrieved from "https://wiki.allizom.org/index.php?title=Releases/Firefox_72/Test_Plan&oldid=1222578"