QA/LZMA Support for Updater

From MozillaWiki
Jump to navigation Jump to search

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
6/7/2017 1.0 Abe Masresha Created first draft
6/13/2017 1.1 Abe Masresha Added link to automation tests and test cases
8/10/2017 1.2 Abe Masresha added signoffs date and other links
8/14/2017 1.3 Abe Masresha adding more information to the test plan

Overview

Purpose

The purpose of this work is to change the update file format using LZMA compression because LZMA compression provides more file compression benefit over the current bzip2 compression.

Scope

  • Testing will focus on:
Stability of the browser with this work.
Whether browser update is applied successfully
If the browser update is in the desired file format
  • How testing will be performed:
Manual
  • Automation is covered by the developer

Ownership

Mozilla Development

Matt Howell

QA Eng. Softvision Las Vegas
Stefan Georgiev - Team Lead
Abe Masresha - Test Lead
Justin Williams - Tester
Grover Wimberly IV - Tester
Kanchan Kumari - Tester

Testing summary

Scope of Testing

In Scope

This feature will be tested manually on Windows, Mac, and Linux operating systems.

Out of Scope

Mobile devices are not affected by this work so this testing does not include them

Automation is handled by the developer
Automated tests:

unit_base_updater
unit

Risk Analysis

  • Update may not be applied so that the user may not get browser updates
  • May behave differently in different operating systems
  • Update may not be applied successfully if the browser has add-on
  • May behave differently if update download is interrupted by no internet connection
  • The update file may not be in desired MAR file format

Requirements for testing

Environments

Windows 10 64bits
Windows 7 32bits
Ubuntu 16.04 32 and 64bits
MacOSX 10.12


Channel dependent settings (configs) and environment setups

Nightly

Firefox Nightly 56.0a1, Test environment setting is provided here
This patch was pushed to Mozilla-central on 7-31-2017, so all older Nightly should update through this update watershed.

Beta

Beta 56.0b3, Test environment setting is provided here
Feature was backed out Bug 1387231 so the new update watershed of beta will be 56b3.
To test on 56b3, the next beta build (56b4) should be avialable as this is an update related work.

Test Strategy

Test Objectives

  • Verify the update is in MAR file format. More information is available here
  • Verify update is applied successfully
  • Verify browser is stable and does not crash with this feature
  • Verify browser does not crash while updating
  • Verify update is applied when the browser has an add-on
  • Verify Firefox is responsive if update download is interrupted by no internet connection

Builds

This section should contain links for builds with the feature -

  • Links for Nightly build: Nightly build
  • Links for Beta build: use beta build 56b3

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 5/30/2017
Study documentation/specs received from developers 6/7/2017 8/14/2017
QA - Test plan creation 6/7/2017 8/14/2017
QA - Test cases/Env preparation 7/31/2017 8/1/2017
QA - Nightly Testing 8/1/2017 8/1/2017
QA - Beta Testing
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail
Test case execution TestRail
Bugs management Bugzilla

Status

Overview

Landed on Nightly on: 7/31/2017
Merged to Beta on: Beta 56b3 will have the patch so start the testing when 56b4 is ready.

References

Testcases

Test Areas

Test Areas Covered Details
Private Window Yes user should be able to get update from private window as well
Multi-Process Enabled Yes user should be able to get update from e10s window
Multi-process Disabled Yes user should be able to get update from Non e10s window as well
Theme (high contrast) Yes high theme shouldnot affect the update
Install/Upgrade
Feature upgrades /Browser update Yes Will be covered on browser updates
Add-ons
Testing with existing/popular addons Yes Feature functionality will be tested with popular addons
Web Compatibility
Survey of popular sites for compatibility Yes Firefox will be tested for stability with popular sites
Interoperability
Interaction of this feature with other browser features Yes Feature will be tested for its interaction with other browser features

Test suite

Full Test suite - test rail
Smoke Test suite -not applicable in this case
Regression Test suite - not applicable in this case

Bug Work

Bug No Summary Status Firefox Version
None in pre-beta testing cycle 56

Sign off

Criteria

Checklist

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools)
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed

Results

Nightly testing

List of OSes that will be covered by testing: see 'Environments' section

  • Link for the tests run


Merge to Beta Sign-off
List of OSes that will be covered by testing

    • Link for the tests run
    • Full Test suite

Checklist

Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [DONE]
Testing Infrastructure setup [DONE]
Test Plan Creation [DONE]
Test Cases Creation [DONE]
Automation Coverage [DONE] Covered by developer
Performance Testing [DONE] Covered by developer
All Defects Logged N/A No bugs found on pre-beta signoff
Critical/Blockers Fixed and Verified N/A
QA mid-Nightly Signoff [MISSED] Missed (test build was not ready)
QA pre-Beta Signoff [DONE] Email sent on 8/1/2017
QA Beta - Full Testing IN PROGRESS
QA pre-Release Signoff IN PROGRESS Email to be sent
Retrieved from "https://wiki.allizom.org/index.php?title=QA/LZMA_Support_for_Updater&oldid=1178352"