QA/Sync/Test Plan

Overview

Outline Quality Assurance plan to adapt to structured development cycles for Sync client and Sync Servers. Both parts of Sync have moved to a weekly cycle of fixes and pushes. This plan shows how QA will handle those weekly trains.

Client

Weekly client pushes of s-c to m-c

Sync client fixes are made on the services-central branch. Once per week, on Monday, those fixes will be handed off to QA for testing

Testing strategy

• Verify fixes per bug STR's
• Run the set of manual smoketest+plus cross platforms to ensure basic Sync functionality has not regressed due to the new fixes.
• Again verify fixes and smoketest against the Wednesday m-c build to ensure no merge issues regressed Sync on Nightly channel

Schedule

• Builds to QA: Monday
• QA sign-off: EOD Tuesday
• s-c push to m-c: Tuesday EOD

Channel progression

Being a built into Firefox service, Sync will follow along with FxN

Schedule

The schedule follows the FxN schedule through the channel progression of releases.

Testing strategy

• Nightly - See the s-c to m-c push plan above.
• Aurora - Run Sync Basic Functional Tests post merge.
  • note: handle specific direct landing to Aurora on a case by case basis.
• Beta - Run Sync smoketests post build.
• Release
  • Run Sync smoketests post build.
  • Upgrade testing when available
  • Major Update testing when available
  • Locale testing - install and quick spot check a few Tier1 locales

Servers

Sync server fixes are made on the staging server(s). Once per week, on Wednesday, those fixes will be handed off to QA for testing. Testing for Sync server, registration server, PAKE server and Account Portal can be required for any or all of them depending on fixes going in that week. Note: per the release process docs, Crossweave and load testing will be performed by Server Ops as necessary. Testing approach to each piece is outlined below.

Sync server

• Staging server is https://stage-auth.services.mozilla.com/
• Verify fixes per bug STR's
• Run the set of manual smoketest+plus cross platforms to ensure basic Sync functionality has not regressed due to the new fixes.

Registration Servers

• Staging server is https://stage-auth.services.mozilla.com/
• Verify fixes per bug STR's
• Run the set of manual smoketests cross platforms to ensure basic account registration functionality has not regressed due to the new fixes.

PAKE Server

• Update about:config (restart required) services.sync.jpake.serverURL to https://stage-setup.services.mozilla.com/
• Verify fixes per bug STR's
• Run manual easy setup tests across devices to ensure functionality has not regressed due to the new fixes.

Account Portal

• Stage Account Portal is https://stage-account.services.mozilla.com/
• Verify fixes per bug STR's
• Run the set of manual test to ensure account management functionality has not regressed due to the new fixes.

Schedule

• Builds to QA: Wednesday
• QA sign-off: EOD Friday
• Push from stage to production: During maintenance window on Monday, 3-5pm Pacific time. (QA to be on hand for live spot checks to help verify the push was a success)