QA/BrowserID/TestPlan: Difference between revisions

From MozillaWiki
< QA‎ | BrowserID
Jump to navigation Jump to search
Line 3: Line 3:


= Strategy =
= Strategy =
Add the BrowserID QA test strategy here...
Create a consistent and repeatable set of tests for qualifying and releasing the weekly builds and deployments for BrowserID. The focus needs to be on quality, but also on providing accurate feedback on successes and issues to Dev and PM so that Production releases provide a useful product for the Mozilla and development communities.
 
Ad Hoc testing to define test paths on the client and server side.
Create useful client-side and server-side test cases that can be imported into Litmus for repeatable manual testing.
Create automated client and server tests that can more accurately cover the full functionality of the UI, the API, and the main roles (PIA, RP, IP) and various flows.
Emphasis on security and privacy issues that might come up across OS, browsers, accounts, and emails on the various clients tested against the server.
Identifying and tracking issues in GitHub and Bugzilla.
Provide useful metrics to Dev and PM.


= Scope of Testing =  
= Scope of Testing =  

Revision as of 19:16, 2 August 2011

Overview

This test plan covers the general weekly testing that will happen against a built, unit-tested, and deployed BrowserID product in the Beta Test environment. The goal is to ensure a defined/consistent amount of quality and usability in the server-side and client side portions of the BrowserID product. Since this is a public-facing product, we need to make sure that the Mozilla community can access, test, and develop around a solid product.

Strategy

Create a consistent and repeatable set of tests for qualifying and releasing the weekly builds and deployments for BrowserID. The focus needs to be on quality, but also on providing accurate feedback on successes and issues to Dev and PM so that Production releases provide a useful product for the Mozilla and development communities.

Ad Hoc testing to define test paths on the client and server side. Create useful client-side and server-side test cases that can be imported into Litmus for repeatable manual testing. Create automated client and server tests that can more accurately cover the full functionality of the UI, the API, and the main roles (PIA, RP, IP) and various flows. Emphasis on security and privacy issues that might come up across OS, browsers, accounts, and emails on the various clients tested against the server. Identifying and tracking issues in GitHub and Bugzilla. Provide useful metrics to Dev and PM.

Scope of Testing

Add the BrowserID QA scope of testing here...

  • Client-side: what will and will not be tested
  • Server-side: what will and will not be tested

General Test Information

Links and Documentation

Weekly Test Schedules

  • Thursdays: deployment to Production, Beta (QA), and Dev
  • Thursdays/Fridays: open testing and experimentation by Dev, QA, and community
  • Following week:
    • Monday - Wednesday: QA testing and sign off of current deployment

Weekly Meetings

Weekly meeting notes are kept on the following EtherPad site:

Posting Feedback

Bugs and Open Issues

Ops Bugs

  • Found in Bugzilla
    • Classification: Other
    • Product: mozilla.org
    • Component: Server Operations: Labs

BrowserID Client/Server Bugs/Issues

Client/Server Test Environment

Beta test environment

  • TBD

Prod test environment

Supported OS and Browsers

Operating Systems

  • PC: Win 7, WinXP, Vista (in priority order)
  • Mac 10.6, 10.7, 10.5
  • Linux: Fedora 14/15, EL5/6, Ubuntu

Browsers

  • Minimal support would have to include a native browser plus Firefox
    • Windows: IE 8, FF 4/5, Google Chrome latest
    • Mac: Safari latest, FF 4/5, Google Chrome latest
    • Linux: FF 3/4/5, Google Chrome latest


Major Test Areas

Code Verification

Verify non-QA-testable fixes/code changes by direct inspection (as required)

Installation/Deployment Verification

QA needs access to Beta test environment in order to verify the deployment to server side (PIA/IP) and client side (RP). It would be helpful for QA to verify the weekly deployment to the server, along with any client-side changes (JavaScript library, JavaScript API). For early trains, this should be covered by information given in the BrowserID deployment ticket (Bugzilla). This would amount to a delta from the previous train that QA can record as part of their testing.

For subsequent trains, the Beta test environment or a QA environment (deployed with the same build) could be used to verify server deployments and client-side changes.

Sanity/Acceptance/Smoke

Small, repeatable set of tests with known, good, expected results. Manual and automated testing on client-side to pass a minimal level of acceptance without which, QA testing of BrowserID can not proceed. This will probably be a very small subset of the basic functional tests or some automated smoke test: TBD

Bug Verification

Manual testing of bugs/issues resolved for this weekly cycle of testing. Test cases generated during this testing can be moved to an automation tool for bug regression (see below).

Basic Functional

Manual and automated testing on the client and the server to verify basic functionality of BrowserID:

  • Creating an account first
  • Creating an account inline (at first use)
  • Creating multiple accounts
  • Deleting one or more accounts
  • Adding additional emails
  • Deleting one or more emails (maintaining the account though)
  • Forgotten account information - mail, passwords
  • Leaving/returning to sites
  • Browser restart after creation of identity

UI

Manual and automated testing on the client and the server to cover all aspects of the current UI:

  • HTML pop-ups
  • Account creation UI
  • Sign-in UI
  • Add a new email address
  • This is not me
  • Is there anything available inside FF desktop or mobile yet?
  • Stay logged in vs. always log out

Bug Regression

Manual and automated testing of Verified/Closed bugs and issues from previous cycles of testing. Ideally, this would be in the form of a quick automated test/script to check key fixes/changes/updates. QA to work with Dev and PM to come up with a list of bugs/issues candidates to from the test cases for manual and automated testing.

OS/Browser integration

Manual testing/verification of the required OS and browser configurations:

  • Verify access and use across browsers on same OS, different OS
  • Verify access and use across OS platforms, same browser, different browser
  • This includes account creation/update/deletion and email addition/deletion
  • OS/Browser-specific local storage verification
  • Browser preferences, esp for privacy/security
  • Browser synchronization - same platform, across platforms

Server-side Testing

  • Verify some UI elements and actions only found on the server:
  • Log inspection to verify accurate information
    • Verify content of web server access logs
    • Verify content of web server error logs
    • Verify content of BrowserID logs
  • What other server-specific tests are required?

Ad Hoc

  • General ease of use
  • Email and Password character compatibility
  • Valid vs. invalid email formats
  • Different accounts using same email/password combos


Areas Not Covered or In Development

Server and Client Automation

Currently, QA has not automation configured for the weekly BrowserID testing. TBD: Client-side automation TBD: Server-side automation, primarily to cover the API

Document Signing

Not much information here, could use help from PM and Dev if specific QA work is needed.

Security: VEP, VES, VEC

QA could use some help from Dev and PM on testable use cases. Valid/invalid public key Security/Crypto checks REF: https://wiki.mozilla.org/Identity/Verified_Email_Protocol (is this the more current link?) https://wiki.mozilla.org/Labs/Identity/VerifiedEmailProtocol

FLOWS/END-TO-END

Verifying each step in the various flows and also the complete End-To-End flow. This testing will need to be done manually, if possible, for a few weekly trains to get the required tests and tools scoped out. Then, this can be moved over to automated E2E testing (alongside the automated smoke testing). This seems like a better fit for server-side automation, where the appropriate actions and verifications could be tested per flow. Important flows to test/validate:

  • Certificate Provisioning: 8 steps to verify full certificate provisioning path
  • Assertion Generation: 4 steps to verify assertion generation path
  • Assertion Verification: 6 steps to verify assertion verification path

QA could use some help from PM and Dev here to better define use cases for each step of these flows. Provide ideas/cases for testing each of the steps, plus the included assumptions and requirements. If applicable we should design both client-side and server-side tests. QA could also leverage off the design/purpose/coverage of the unit tests. REF: http://lloyd.io/how-browserid-works

FireFox Add-On

For Firefox users with the addon, the addon flow needs tested. Test design: TBD


Setting Up Test Emails

Email-side (PIA)

  • This is the Primary Identity Authority (PIA) for all the transactions.
  • Examples for testing: Mozilla, Yahoo, Google, ISPs
  • Also available for testing are sites like Mailinator

Using Internal Email Accounts

  • If secure enough to do so: <mozilla name>+tag@mozilla.com
  • Otherwise: Create emails at www.mailinator.com
  • Do we have other tested (re: reliable) methods for creating test/dummy email accounts?

Open Questions/Issues

1. How do we verify this statement: "Unlike other sign-in systems, BrowserID does not leak information back to any server (not even to the BrowserID servers) about which sites a user visits." Could use some help from Dev and PM on testable use cases. 2. How does this site/technology fit in? Or is it not applicable to our weekly testing? http://people.mozilla.com/~faaborg/files/projects/firefoxAccount/index.html

Retrieved from "https://wiki.allizom.org/index.php?title=QA/BrowserID/TestPlan&oldid=335612"