QA/LZMA Support for Updater: Difference between revisions

From MozillaWiki
< QA
Jump to navigation Jump to search
 
(80 intermediate revisions by 2 users not shown)
Line 1: Line 1:
'''Approvals Required / Received'''
The following individuals are required to/have approved this Test Plan:
{| class="wikitable"
|-
! Name !! Title !! Department !! Approval Date !! Method
|-
| RyanVM  || QA Reviewer ||Product Integrity || n/a||    Email
|-
|Matt Howell  || Software Engineer || Engineering || 6/14/2017 ||    Email
|-
| David Durst || EPM || Product Management || n/a ||  Email
|}
'''Revision History'''
'''Revision History'''


Line 9: Line 24:
| 6/7/2017 || 1.0 || Abe Masresha || Created first draft
| 6/7/2017 || 1.0 || Abe Masresha || Created first draft
|-
|-
| || 1.1 || . ||
| 6/13/2017 || 1.1 || Abe Masresha || Added link to automation tests and test cases
|-
| 8/10/2017 || 1.2 || Abe Masresha || added signoffs date and other links
|-
| 8/14/2017 || 1.3 || Abe Masresha || adding more information to the test plan
|-
|-
|}
|}
Line 15: Line 34:
= Overview =
= Overview =
== Purpose ==
== Purpose ==
The purpose of this work is to change the update file format.
The purpose of this work is to change the update file format using LZMA compression because LZMA compression provides more file compression benefit over the current bzip2 compression.  


== Scope ==
== Scope ==
Line 21: Line 40:


::: Stability of the browser with this work.
::: Stability of the browser with this work.
::: Whether browser update is applied or not
::: Whether browser update is applied successfully
::: If the update is in the desired file format
::: If the browser update is in the desired file format


* How testing will be performed:
* How testing will be performed:
:::  Manual
:::  Manual


*Automation is covered by the developer
*Automation is covered by the developer  
::: [Link to automation test script will be provided here]
 
 


== Ownership ==
== Ownership ==
Line 36: Line 52:
  Matt Howell<br />
  Matt Howell<br />


'''QA Eng. Softvision Las Vegas'''<br />
'''QA Eng. Softvision Las Vegas - Nightly testing'''<br />
[mailto:stefan.georgiev@softvision.com Stefan Georgiev - Team Lead ]<br />
[mailto:amasresha@softvision.com Abe Masresha]<br />
[mailto:amasresha@softvision.com Abe Masresha - Test Lead]<br />
 
[mailto:jwilliams@softvision.com Justin Williams - Tester]<br />
'''QA - Beta testing'''<br />
[mailto:gwimberly@softvision.com Grover Wimberly IV - Tester]<br />
[mailto:petruta.rasa@softvision.ro Petruta Rasa]<br />
[mailto:kkumari@softvision.com Kanchan Kumari - Tester]<br />
 
= Testing summary =  
= Testing summary =  
== Scope of Testing ==
== Scope of Testing ==
=== In Scope ===
=== In Scope ===
This feature will be tested manually on Windows, Mac, and Linux operating systems.
This feature will be tested manually on Windows, Mac, and Linux operating systems. <br>
 
Automation is handled by the developer<br />
Automated tests: <br>
::[https://hg.mozilla.org/mozilla-central/file/tip/toolkit/mozapps/update/tests/unit_base_updater unit_base_updater]
::[https://hg.mozilla.org/mozilla-central/file/tip/modules/libmar/


=== Out of Scope ===
=== Out of Scope ===
Automation will be handled by the developer<br />
Mobile devices are not affected by this work so this testing does not include them <br>
Mobile devices will not be tested.


tests/unit unit]


= Requirements for testing =
= Requirements for testing =
Line 59: Line 79:
  Ubuntu 16.04 32 and 64bits
  Ubuntu 16.04 32 and 64bits
  MacOSX 10.12
  MacOSX 10.12


== Channel dependent settings (configs) and environment setups ==
== Channel dependent settings (configs) and environment setups ==
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">


=== Nightly ===
=== Nightly ===
<div class="mw-collapsible-content">
Firefox Nightly 56.0a1, [https://docs.google.com/a/mozilla.com/document/d/1BW81CCLvKPsMlFvu1BSIOHXNLzFIKKL7UpBifMrKJI4/edit?usp=sharing Test environment setting is provided here]<br>
Firefox Nightly 56.0a1 <br>
This patch was pushed to Mozilla-central on 7-31-2017, so all older Nightly should update through this update watershed.
</div>


=== Beta ===
=== Beta ===
<div class="mw-collapsible-content">
Beta 56.0b3, [https://docs.google.com/a/mozilla.com/document/d/1BW81CCLvKPsMlFvu1BSIOHXNLzFIKKL7UpBifMrKJI4/edit?usp=sharing Test environment setting is provided here]<br>
Beta 56.0b1
Feature was backed out [https://bugzilla.mozilla.org/show_bug.cgi?id=1387231 Bug 1387231] so the new update watershed of beta will be 56b3. <br>
</div>
To test on 56b3, the next beta build (56b4) should be avialable as this is an update related work.<br>


= Test Strategy =  
= Test Strategy =  
== Risk Analysis ==
== Risk Assessment and Coverage ==
* The update may not be available in desired file format
 
* Update may not be applied so that the user may not get browser updates
{| class="wikitable"
* May behave differently in different operating systems
|-
! ID !! Description / Threat Description !! Covered by Test Objective !!  Magnitude !! Probability !! Priority !! Impact Score
|-
| RAC-1 ||  Update may not be applied to the browser || TO-1 ||  3-High || 2-Possible || 3-High ||18
|-
| RAC-2 || Update may not be applied successfully if the browser has add-on || TO-2 || 3-High || 2-Possible || 2-Medium || 12
|-
| RAC-3 || The update file may not be in desired MAR file format || TO-3 || 2-Moderate || 1-Unlikely || 2-Medium || 4
|-
| RAC-4 || May behave differently if update download is interrupted by no internet connection || TO-4 || 2-Moderate || 1-Unlikely || 2-Medium || 4
|-
| RAC-5 || May behave differently in different operating systems || TO-5 || 2-Moderate || 1-Unlikely || 2-Medium || 4
 
|}
 
'''Values:'''
 
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High'''
 
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain'''
 
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High'''
 
'''Impact Score Breakdown:'''
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.


== Test Objectives ==
== Test Objectives ==
*Verify update is available in desired file format
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.
*Verify update is applied successfully
{| class="wikitable"
*Verify browser is stable and does not crash with this feature
|-
*Verify browser does not crash while updating
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners
|-
| 1 || Update the browser || Browser should update to its latest build  || update the browser || Manual || RAC-1,RAC-2,RAC-4,RAC-5 || Eng Team
|-
| 2 ||Check the update file format || The update file should be in desired MAR file format || Check if the file is in MAR format. [https://docs.google.com/a/mozilla.com/document/d/1BW81CCLvKPsMlFvu1BSIOHXNLzFIKKL7UpBifMrKJI4/edit?usp=sharing steps are here] || Manual || RAC-3 || Eng Team
 
|}


== Builds ==
== Builds ==
This section should contain links for builds with the feature -  
This section should contain links for builds with the feature -  
* Links for Nightly builds [Will be here when it is ready]
* Links for Nightly build:  [https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/ Nightly build]
* Links for Beta builds [Will be here when it is ready]
* Links for Beta build: use beta build later than [https://archive.mozilla.org/pub/firefox/candidates/56.0b3-candidates/build1/ 56b3]


== Test Execution Schedule ==
== Test Execution Schedule ==
Line 98: Line 147:
|-
|-
| Start project  
| Start project  
|style="text-align:center;" | 5/30/2017  ||  
|style="text-align:center;" | 5/30/2017  || 9/25/2017
|-
|-
| Study documentation/specs received from developers
| Study documentation/specs received from developers
|style="text-align:center;" | 6/7/2017 ||  
|style="text-align:center;" | 6/7/2017 || 8/14/2017
|-
|-
| QA - Test plan creation  
| QA - Test plan creation  
|style="text-align:center;" | 6/7/2017 ||  
|style="text-align:center;" | 6/7/2017 || 8/14/2017
|-
|-
| QA - Test cases/Env preparation  
| QA - Test cases/Env preparation  
|style="text-align:center;" |  ||  
|style="text-align:center;" |7/31/2017 || 8/31/2017
|-
|-
| QA - Nightly Testing  
| QA - Nightly Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | 8/1/2017 || 8/1/2017
|-
|-
| QA - Beta Testing  
| QA - Beta Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | 8/10/2017 || 9/8/2017
|-
|-
| Release Date  
| Release Date  
|style="text-align:center;" | ||  
|style="text-align:center;" | 9/19/2017 || 9/26/2017
|}
|}


Line 127: Line 176:
| Test plan creation || Mozilla wiki
| Test plan creation || Mozilla wiki
|-
|-
| Test case creation || [https://testrail.stage.mozaws.net/index.php TestRail]/ Google docs
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/suites/view/1110&group_by=cases:section_id&group_order=desc TestRail]  
|-
|-
| Test case execution || [https://testrail.stage.mozaws.net/index.php TestRail]
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/plans/view/5274 TestRail]
|-
|-
| Bugs management || Bugzilla
| Bugs management || [https://bugzilla.mozilla.org/ Bugzilla]
|}
|}


= Status =  
= Status =  
== Overview ==
== Overview ==
Released to Nightly on: []
Landed on Nightly on: 7/31/2017 <br>
Merged to Beta on: []
Merged to Beta on: Beta 56b3 will have the patch so start the testing when 56b4 is ready.


= References =
= References =
* List and links for specs
* [https://bugzilla.mozilla.org/show_bug.cgi?id=641212 Main bug:641212] <br>
  List and links for available specs - documents, user stories, specifications
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1303172  Meta bug:1303172]
* Meta bug
* [https://docs.google.com/a/mozilla.com/document/d/1BW81CCLvKPsMlFvu1BSIOHXNLzFIKKL7UpBifMrKJI4/edit?usp=sharing Additional information to QE]
 
= Testcases =  
= Testcases =  
== Test Areas ==
== Test Areas ==
Line 151: Line 200:
|-
|-
| Private Window  
| Private Window  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes|| user should be able to get update from private window as well
|-
|-
| Multi-Process Enabled  
| Multi-Process Enabled  
|style="text-align:center;" |  ||  
|style="text-align:center;" |  Yes|| user should be able to get update from e10s window
|-
|-
| Multi-process Disabled  
| Multi-process Disabled  
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes || user should be able to get update from Non e10s window as well
|-
|-
| Theme (high contrast)  
| Theme (high contrast)  
|style="text-align:center;" |  ||  
|style="text-align:center;" |  Yes|| high theme shouldnot affect the update
|-
| '''UI'''
||  ||
|-
| Mouse-only operation 
|style="text-align:center;" |  ||
|-
| Keyboard-only operation 
|style="text-align:center;" |  ||
|-
| Display (HiDPI)
|style="text-align:center;" | ||
|-
| Interaction (scroll, zoom)
|style="text-align:center;" | ||
|-
| Usable with a screen reader 
|style="text-align:center;" |  || e.g. with NVDA
|-
| Usability and/or discoverability testing 
|style="text-align:center;" |  || Is this feature user friendly
|-
| RTL build testing 
|style="text-align:center;" |  ||
|-
| '''Help/Support'''
||  ||
|-
| Help/support interface required 
|style="text-align:center;" |  || Make sure link to support/help page exist and is easy reachable.
|-
| Support documents planned(written) 
|style="text-align:center;" |  || Make sure support documents are written and are correct.
 
|-
|-
| '''Install/Upgrade'''  
| '''Install/Upgrade'''  
||  ||  
||  ||  
|-
|-
| Feature upgrades/downgrades data as expected 
| Feature upgrades /Browser update
|style="text-align:center;" | ||
|style="text-align:center;" | Yes || Will be covered on browser updates
|-
|-
| Does sync work across upgrades 
|style="text-align:center;" |  ||
|-
|-
| Requires install testing   
| ''' Add-ons '''  
|style="text-align:center;" |  || separate feature/application installation needed (not only Firefox)
|-
| Affects first-run or onboarding   
|style="text-align:center;" |  || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|-
| Does this affect partner builds? Partner build testing 
|style="text-align:center;" |  || yes/no options, add comment with details about who will lead testing
 
|-
| ''' Enterprise '''  
||  ||  Raise up the topic to developers to see if they are expecting to work different on ESR builds
|-
| Enterprise administration 
|style="text-align:center;" |  ||
|-
| Network proxies/autoconfig 
|style="text-align:center;" |  ||
|-
| ESR behavior changes 
|style="text-align:center;" |  ||
|-
| Locked preferences 
|style="text-align:center;" |  ||
 
|-
| ''' Data Monitoring '''  
||  ||
|-
| Temporary or permanent telemetry monitoring 
|style="text-align:center;" |  || List of error conditions to monitor
|-
| Telemetry correctness testing 
|style="text-align:center;" |  ||
|-
| Server integration testing 
|style="text-align:center;" |  ||
|-
| Offline and server failure testing 
|style="text-align:center;" |  ||
|-
| Load testing 
|style="text-align:center;" |  ||


|-
| ''' Add-ons '''
||  || If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
|-
| Addon API required? 
|style="text-align:center;" |  ||
|-
| Comprehensive API testing 
|style="text-align:center;" |  ||
|-
| Permissions 
|style="text-align:center;" |  ||
|-
|-
| Testing with existing/popular addons
| Testing with existing/popular addons
|style="text-align:center;" | ||
|style="text-align:center;" | Yes || Feature functionality will be tested with popular addons
 
|-
| ''' Security '''
||  || Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
|-
| 3rd-party security review 
|style="text-align:center;" |  ||
|-
| Privilege escalation testing
|style="text-align:center;" |  ||
|-
| Fuzzing 
|style="text-align:center;" |  ||  


|-
|-
| ''' Web Compatibility '''  
| ''' Web Compatibility '''  
||  || depends on the feature
|-
| Testing against target sites 
|style="text-align:center;" |  ||
|-
|-
| Survey of many sites for compatibility   
| Survey of popular sites for compatibility   
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes || Firefox will be tested for stability with popular sites


|-
|-
| ''' Interoperability  '''  
| ''' Interoperability  '''  
||  || depends on the feature
|-
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. 
|style="text-align:center;" |  ||
|-
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS 
|style="text-align:center;" |  ||
|-
|-
| Interaction of this feature with other browser features   
| Interaction of this feature with other browser features   
|style="text-align:center;" | ||  
|style="text-align:center;" | Yes || Feature will be tested for its interaction with other browser features
|}
|}


== Test suite ==
== Test suite ==
Full Test suite - Link to test rail - testcases should be added under Firefox Desktop project [https://testrail.stage.mozaws.net/index.php?/suites/overview/17 link]
Full Test suite - [https://testrail.stage.mozaws.net/index.php?/suites/view/1110&group_by=cases:section_id&group_order=desc test rail] <br>
Smoke Test suite - Link with the tests - if available/needed.
Regression Test suite - Link with the tests - if available/needed.


= Bug Work =
= Bug Work =
Meta bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345 - bug summary]
{| class="wikitable" style="width:50%"
 
|-
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
! Bug No !! Summary !! Status !! Firefox Version
====== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=12345 12345] )======
|-
 
|None in pre-beta testing cycle || ||  || 56
<div class="mw-collapsible-content">
|-
<bugzilla>
|-
    {
|None during Beta testing cycle || ||  || 56
        "blocks":[12345],
|-
        "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
|}
    }
</bugzilla>
 
</div>
</div>
 
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
====== Bug fix verification ======
<div class="mw-collapsible-content">
<bugzilla>
    {
        "blocks":[12345],
        "resolution":"FIXED",
        "include_fields": "id, priority, component, assigned_to, summary, status, resolution, target_milestone"
    }
</bugzilla>
</div>
</div>


= Sign off =
= Sign off =
Line 341: Line 256:
Checklist
Checklist
* All test cases should be executed
* All test cases should be executed
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
* Has sufficient automated test coverage (as measured by code coverage tools)
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed  


== Results ==
== Results ==
'''Nightly testing'''<br />
'''Nightly testing'''<br />
List of OSes that will be covered by testing:  see 'Environments' section <br />
**Links:
** [https://testrail.stage.mozaws.net/index.php?/suites/view/1110&group_by=cases:section_id&group_order=desc Full Test suite]<br>
**[https://testrail.stage.mozaws.net/index.php?/plans/view/5274 Runs and Results]<br>
** Daily Smoke -not applicable to the pre-beta test cycle as the test build was landed on the last day of the pre-beta sign off <br>
** Regression Test suite, No regression bugs were found <br />


List of OSes that will be covered by testing<br />
'''Preliminary status report on Beta'''<br />
*Link for the tests run
Preliminary status report analysis shows that updating from Firefox 55.0bX, updates first to 56.0b3 and then to latest beta available on the channel. Also, update.mar files are correct and have the right format. Preliminary recommendation is to "ship it". <br />
** Full Test suite, link to TestRail - Tests Runs and Results [https://testrail.stage.mozaws.net/index.php?/runs/overview/17 link]
** Daily Smoke, if needed/available
** Regression Test suite, if needed/available
<br />


'''Merge to Beta Sign-off'''<br />
'''Pre-Release Sign-off - GREEN 2017-09-06''' <br />
List of OSes that will be covered by testing<br />
*List of OSes covered by testing: Windows 7 64-bit, Windows 10 64-bit, Ubuntu 16.04 64-bit and Mac OS X 10.11<br />
*Link for the tests run
** Link for the tests run -  [https://testrail.stage.mozaws.net/index.php?/plans/view/5274 link]
** Full Test suite
** Regression Test suite, No regression bugs were found <br />


== Checklist ==
== Checklist ==
Line 365: Line 282:
|-
|-
|  Testing Prerequisites (specs, use cases)  
|  Testing Prerequisites (specs, use cases)  
| style="text-align:center;" |   
| style="text-align:center;" |  {{mdone|}}
| style="text-align:center;" |  
| style="text-align:center;" |  
|-
|-
|  Testing Infrastructure setup  
|  Testing Infrastructure setup  
|style="text-align:center;" |   ||  
|style="text-align:center;" | {{mdone|}}  ||  
|-
|-
|  Test Plan Creation  
|  Test Plan Creation  
| style="text-align:center;" |   ||  
| style="text-align:center;" | {{mdone|}}  ||  
|-
|-
|  Test Cases Creation  
|  Test Cases Creation  
|style="text-align:center;" |   ||  
|style="text-align:center;" | {{mdone|}}  ||  
|-
|-
|  Automation Coverage ||
|  Automation Coverage  
|style="text-align:center;" |  
|style="text-align:center;" |{{mdone|}}
|style="text-align:center;" | Covered by developer
|-
|-
|  Performance Testing  
|  Performance Testing  
|style="text-align:center;" | ||
|style="text-align:center;" |{{mdone|}}
|-
|style="text-align:center;" |  Covered by developer
|  All Defects Logged || ||
|-
|  Critical/Blockers Fixed and Verified || ||
|-
|  Metrics/Telemetry||
|style="text-align:center;" |
|-
|  Basic/Core functionality Nightly testing
|style="text-align:center;" | 
|style="text-align:center;" |   
|-
|-
|  QA mid-Nightly Signoff||  
|  All Defects Logged 
|style="text-align:center;" | Email to be sent
|style="text-align:center;" |{{N/A|}}
|style="text-align:center;" | No bugs found on pre-beta signoff
|-
|-
|  QA Nightly - Full Testing
|  Critical/Blockers Fixed and Verified
|style="text-align:center;" | ||  
|style="text-align:center;" |{{N/A|}}||
|-
|  QA mid-Nightly Signoff
|style="text-align:center;" |{{Missed |}}
|style="text-align:center;"| Missed (test build was not ready)
|-
|-
|  QA pre-Beta Signoff||  
|  QA pre-Beta Signoff
|style="text-align:center;"| Email to be sent  
|style="text-align:center;" |{{mdone|}}
|style="text-align:center;"| Email sent on 8/1/2017
|-
|-
|  QA Beta - Full Testing
|  QA Beta - Full Testing
|style="text-align:center;" | ||  
|style="text-align:center;" |{{mdone|}}
|style="text-align:center;" | 
|-
|-
|  QA pre-Release Signoff ||  
|  QA pre-Release Signoff  
|style="text-align:center;" | Email to be sent  
|style="text-align:center;" |{{mdone|}}
|style="text-align:center;" | Email sent on 9/6/2017
|}
|}

Latest revision as of 09:54, 6 September 2017

Approvals Required / Received

The following individuals are required to/have approved this Test Plan:

Name Title Department Approval Date Method
RyanVM QA Reviewer Product Integrity n/a Email
Matt Howell Software Engineer Engineering 6/14/2017 Email
David Durst EPM Product Management n/a Email

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
6/7/2017 1.0 Abe Masresha Created first draft
6/13/2017 1.1 Abe Masresha Added link to automation tests and test cases
8/10/2017 1.2 Abe Masresha added signoffs date and other links
8/14/2017 1.3 Abe Masresha adding more information to the test plan

Overview

Purpose

The purpose of this work is to change the update file format using LZMA compression because LZMA compression provides more file compression benefit over the current bzip2 compression.

Scope

  • Testing will focus on:
Stability of the browser with this work.
Whether browser update is applied successfully
If the browser update is in the desired file format
  • How testing will be performed:
Manual
  • Automation is covered by the developer

Ownership

Mozilla Development

Matt Howell

QA Eng. Softvision Las Vegas - Nightly testing
Abe Masresha

QA - Beta testing
Petruta Rasa

Testing summary

Scope of Testing

In Scope

This feature will be tested manually on Windows, Mac, and Linux operating systems.

Automation is handled by the developer
Automated tests:

unit_base_updater
[https://hg.mozilla.org/mozilla-central/file/tip/modules/libmar/

Out of Scope

Mobile devices are not affected by this work so this testing does not include them

tests/unit unit]

Requirements for testing

Environments

Windows 10 64bits
Windows 7 32bits
Ubuntu 16.04 32 and 64bits
MacOSX 10.12

Channel dependent settings (configs) and environment setups

Nightly

Firefox Nightly 56.0a1, Test environment setting is provided here
This patch was pushed to Mozilla-central on 7-31-2017, so all older Nightly should update through this update watershed.

Beta

Beta 56.0b3, Test environment setting is provided here
Feature was backed out Bug 1387231 so the new update watershed of beta will be 56b3.
To test on 56b3, the next beta build (56b4) should be avialable as this is an update related work.

Test Strategy

Risk Assessment and Coverage

ID Description / Threat Description Covered by Test Objective Magnitude Probability Priority Impact Score
RAC-1 Update may not be applied to the browser TO-1 3-High 2-Possible 3-High 18
RAC-2 Update may not be applied successfully if the browser has add-on TO-2 3-High 2-Possible 2-Medium 12
RAC-3 The update file may not be in desired MAR file format TO-3 2-Moderate 1-Unlikely 2-Medium 4
RAC-4 May behave differently if update download is interrupted by no internet connection TO-4 2-Moderate 1-Unlikely 2-Medium 4
RAC-5 May behave differently in different operating systems TO-5 2-Moderate 1-Unlikely 2-Medium 4

Values:

  • Magnitude: 1- Low , 2-Moderate, 3-High
  • Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
  • Priority: 1 - Low, 2-Medium, 3-High

Impact Score Breakdown:

  • An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
  • An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
  • An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master.

Ref Function Test Objective Evaluation Criteria Test Type RAC Owners
1 Update the browser Browser should update to its latest build update the browser Manual RAC-1,RAC-2,RAC-4,RAC-5 Eng Team
2 Check the update file format The update file should be in desired MAR file format Check if the file is in MAR format. steps are here Manual RAC-3 Eng Team

Builds

This section should contain links for builds with the feature -

  • Links for Nightly build: Nightly build
  • Links for Beta build: use beta build later than 56b3

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 5/30/2017 9/25/2017
Study documentation/specs received from developers 6/7/2017 8/14/2017
QA - Test plan creation 6/7/2017 8/14/2017
QA - Test cases/Env preparation 7/31/2017 8/31/2017
QA - Nightly Testing 8/1/2017 8/1/2017
QA - Beta Testing 8/10/2017 9/8/2017
Release Date 9/19/2017 9/26/2017

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail
Test case execution TestRail
Bugs management Bugzilla

Status

Overview

Landed on Nightly on: 7/31/2017
Merged to Beta on: Beta 56b3 will have the patch so start the testing when 56b4 is ready.

References

Testcases

Test Areas

Test Areas Covered Details
Private Window Yes user should be able to get update from private window as well
Multi-Process Enabled Yes user should be able to get update from e10s window
Multi-process Disabled Yes user should be able to get update from Non e10s window as well
Theme (high contrast) Yes high theme shouldnot affect the update
Install/Upgrade
Feature upgrades /Browser update Yes Will be covered on browser updates
Add-ons
Testing with existing/popular addons Yes Feature functionality will be tested with popular addons
Web Compatibility
Survey of popular sites for compatibility Yes Firefox will be tested for stability with popular sites
Interoperability
Interaction of this feature with other browser features Yes Feature will be tested for its interaction with other browser features

Test suite

Full Test suite - test rail

Bug Work

Bug No Summary Status Firefox Version
None in pre-beta testing cycle 56
None during Beta testing cycle 56

Sign off

Criteria

Checklist

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools)
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed

Results

Nightly testing
List of OSes that will be covered by testing: see 'Environments' section

    • Links:
    • Full Test suite
    • Runs and Results
    • Daily Smoke -not applicable to the pre-beta test cycle as the test build was landed on the last day of the pre-beta sign off
    • Regression Test suite, No regression bugs were found

Preliminary status report on Beta
Preliminary status report analysis shows that updating from Firefox 55.0bX, updates first to 56.0b3 and then to latest beta available on the channel. Also, update.mar files are correct and have the right format. Preliminary recommendation is to "ship it".

Pre-Release Sign-off - GREEN 2017-09-06

  • List of OSes covered by testing: Windows 7 64-bit, Windows 10 64-bit, Ubuntu 16.04 64-bit and Mac OS X 10.11
    • Link for the tests run - link
    • Regression Test suite, No regression bugs were found

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [DONE]
Testing Infrastructure setup [DONE]
Test Plan Creation [DONE]
Test Cases Creation [DONE]
Automation Coverage [DONE] Covered by developer
Performance Testing [DONE] Covered by developer
All Defects Logged N/A No bugs found on pre-beta signoff
Critical/Blockers Fixed and Verified N/A
QA mid-Nightly Signoff [MISSED] Missed (test build was not ready)
QA pre-Beta Signoff [DONE] Email sent on 8/1/2017
QA Beta - Full Testing [DONE]
QA pre-Release Signoff [DONE] Email sent on 9/6/2017
Retrieved from "https://wiki.allizom.org/index.php?title=QA/LZMA_Support_for_Updater&oldid=1179946"