QA/BrowserID/TestPlan

From MozillaWiki
Jump to navigation Jump to search

Back To ==> Mozilla BT Services


Overview

This test plan covers the general weekly testing that will happen against a built, unit-tested, and deployed BrowserID product in the Beta Test environment. The goal is to ensure a defined/consistent amount of quality and usability in the server-side and client side portions of the BrowserID product. Since this is a public-facing product, we need to make sure that the Mozilla community can access, test, and develop around a solid product.

Strategy

Create a consistent and repeatable set of tests for qualifying and releasing the weekly builds and deployments for BrowserID. The focus needs to be on quality, but also on providing accurate feedback on successes and issues to Dev and PM so that the Production releases provide a useful product for the Mozilla and development communities.

  • Ad Hoc testing to define test paths on the client and server side.
  • Create useful client-side and server-side test cases that can be imported into Litmus for repeatable manual testing.
  • Create automated client and server tests that can more accurately cover the full functionality of the UI, the API, and the main roles (PIA, RP, IP) and various flows.
  • Emphasis on security and privacy issues that might come up across OS, browsers, accounts, and emails on the various clients tested against the server.
  • Identifying and tracking issues in GitHub and Bugzilla.
  • Provide useful metrics to Dev and PM.

Scope of Testing

On the Client-side, testing will cover the basic functionality and UI, accounts and emails, interaction with the Server, security/privacy, and usability/compatibility across OS and browsers.

On the Server-side, testing will start with basic functionality, support for multiple client sites, user security/privacy, communication with various clients, then move on to information handling and storage, security/privacy of information, information persistence across deployments, logging.

General Test Information

Links and Documentation

Weekly Test Schedules

  • Thursdays: deployment to Production, Beta (QA), and Dev
  • Thursdays/Fridays: open testing and experimentation by Dev, QA, and community
  • Following week: Monday - Wednesday: QA testing and sign off of current deployment

Weekly Meetings

Weekly meeting notes are kept on the following Mozilla wiki site:

Posting Feedback

Bugs and Open Issues

Ops Bugs in Bugzilla

  • Classification: Other
  • Product: mozilla.org
  • Component: Server Operations: Labs
  • Note: This includes the weekly release deployment ticket.

BrowserID Client/Server Bugs/Issues

Client/Server Test Environment

Beta test environment

Prod test environment

Supported OS and Browsers

Operating Systems/Platforms

  • PC: Windows 7 (64bit/32bit), Windows XP (32bit), Vista (in priority order)
  • Mac 10.6, 10.7, 10.5
  • Linux: Fedora 14/15, EL5/6, Ubuntu 10/11
  • Mobile: Android (latest), iOS (latest)

Browsers

  • Minimal support would have to include a native browser plus Firefox
    • Windows: IE 8/9, FireFox 6, Aurora, Beta, Google Chrome (latest), Opera (latest)
    • Mac: Safari 5 or newer, FireFox 6, Aurora, Beta, Google Chrome (latest), Opera (latest)
    • Linux: FireFox 6, Aurora, Beta, Google Chrome (latest), Opera (latest)

Logging

There are four primary logs that should be monitored during testing: two for BrowserID and two for nginx:

  • Web Server
    • /var/log/nginx/access.log
    • /var/log/nginx/error.log
      • Note: archived logs should also be checked as needed
  • BrowserID (node) logs
    • /home/browserid/var/log/browserid.log
    • /home/browserid/var/log/verifier.log
      • Note: there is no log archiving at this time

Major Test Areas

Code Verification

Verify non-QA-testable fixes/code changes by direct inspection (as required)

Installation/Deployment Verification

Verify the deployment version, processes, and logging in Beta.

  • Processes
    • Make sure node.js is running the following:
      • /home/browserid/code/verifier/run.js
      • /home/browserid/code/browserid/run.js
    • Note: This assumes access to browserid-stage1.vm1.labs.sjc1.mozilla.com. If you do not have SSH access, then get Dev to verify the processes.
  • Verify logging is active
    • Make sure that all logs are up-to-date and getting current information
      • /var/log/nginx/access.log
      • /var/log/nginx/error.log
      • /home/browserid/var/log/browserid.log
      • /home/browserid/var/log/verifier.log
    • Note: This assumes access to browserid-stage1.vm1.labs.sjc1.mozilla.com. If you do not have SSH access, then get Dev to verify that the logs are active.

Sanity/Acceptance/Smoke: TBD

  • Small, repeatable set of tests with known, good, expected results.
  • Manual and automated testing on client-side to pass a minimal level of acceptance without which, QA testing of BrowserID can not proceed.
  • This will probably be a very small subset of the basic functional tests or some automated smoke test.

Bug Verification

  • Manual testing of bugs/issues resolved for this weekly cycle of testing.
  • Dev will list all resolved/closed issues in the deployment ticket for the weekly cycle of testing.
  • Test cases generated during this testing can be moved to an automation tool for bug regression (see below).

Basic Functional

Manual and automated testing on the client and the server to verify basic functionality of BrowserID:

  • Creating an account first
  • Creating an account inline (at first use)
  • Email verification for new accounts and emails
    • Would be useful to have email accounts on various, popular email servers/services
    • Need a method to verify this by direct inspection in server: comparing token in email link to information on server (or wherever this is temporarily stored)
  • Creating multiple accounts
  • Deleting one or more accounts (cancellation)
  • Adding additional emails
  • Deleting one or more emails (maintaining the account though)
  • Forgotten account information - mail, passwords
  • Leaving/returning to sites
  • Browser restart after creation of identity
  • Always logging out from sites vs. never logging out from sites
  • Email and Password character compatibility
  • Valid vs. invalid email formats
  • Different accounts using same email/password combos
  • Shared access to same computer or profiles or accounts
  • Browser settings and preferences, esp. pop-ups, security, privacy
  • FireFox about:config, similar, if any, for other browsers

UI

Manual and automated testing on the client and the server to cover all aspects of the current UI: (yes, this is a bit of an overkill for a Test Plan...)

Bug Regression

OS/Browser integration

Manual testing/verification of the required OS and browser configurations:

  • Verify access and use across browsers on same OS, different OS
  • Verify access and use across OS platforms, same browser, different browser
  • This includes account creation/update/deletion and email addition/deletion
  • OS/Browser-specific local storage verification
  • Browser preferences, esp for privacy/security
  • Browser synchronization - same platform, across platforms

Mobile Testing: TBD

Milestone is completed, so some Mobile specific tests should be added: TBD

  • Sanity/Acceptance
  • Functional/UI
  • OS/Browser

Client-side LocalStorage: TBD

QA will need the local storage information per browser in order to verify accurate creation/update/use/removal of BrowserID data.

  • Possible FireFox example:
    • Users > USER-NAME > Library > Application Support > Firefox > Profiles > PROFILE.default
  • Where is the local storage for BrowserID?
    • Is it just localstore.rdf, or a whole set of files per profile?
  • Local storage verification
    • Per OS, per browser, per user/account
    • Is this also further defined by profile?
      • OS > browser > profile > account/user

Server-side Testing

Need the list of server and services or components that describe the BrowserID architecture to accurately focus on server-side testing.


Areas Not Covered or In Development

Server and Client Automation

Currently, QA has no automation configured for the weekly BrowserID testing.

  • Client-side automation using Selenium is being investigated
  • TBD: Server-side automation, primarily to cover the API

Security: VEP, VES, VEC

QA could use some help from Dev and PM on testable use cases.

Flows/End-To-End

Verifying each step in the various flows and also the complete End-To-End flow. This testing will need to be done manually, if possible, for a few weekly trains to get the required tests and tools scoped out. Then, this can be moved over to automated E2E testing (alongside the automated smoke testing). This seems like a better fit for server-side automation, where the appropriate actions and verifications could be tested per flow. Important flows to test/validate:

  • Certificate Provisioning: 8 steps to verify full certificate provisioning path
  • Assertion Generation: 4 steps to verify assertion generation path
  • Assertion Verification: 6 steps to verify assertion verification path

QA could use some help from PM and Dev here to better define use cases for each step of these flows. Provide ideas/cases for testing each of the steps, plus the included assumptions and requirements. If applicable we should design both client-side and server-side tests. QA could also leverage off the design/purpose/coverage of the unit tests. REF: http://lloyd.io/how-browserid-works

FireFox Add-On

For Firefox users with the addon, the addon flow needs tested. Test design: TBD

Internationalization, Localization

  • Localization (L10n) requirements TBD
  • Internationalization (i10n) requirements TBD

Setting Up Test Emails

Email-side (PIA)

  • This is the Primary Identity Authority (PIA) for all the transactions.
  • Examples for testing: Mozilla, Yahoo, Google, ISPs
  • Also available for testing are sites like Mailinator

Using Internal Email Accounts

  • If secure enough to do so: <mozilla name>+tag@mozilla.com
  • Otherwise: Create emails at www.mailinator.com
  • Do we have other tested (re: reliable) methods for creating test/dummy email accounts?


Back To ==> Mozilla BT Services

Retrieved from "https://wiki.allizom.org/index.php?title=QA/BrowserID/TestPlan&oldid=344202"