QA/BrowserID/TestPlan: Difference between revisions

Latest revision as of 01:29, 3 July 2013

Overview

This test plan covers the general weekly testing that will happen against a built, unit-tested, and deployed BrowserID product in the Stage Test environment. The goal is to ensure a defined/consistent amount of quality and usability in the server-side and RP/client side portions of the BrowserID product. Since this is a public-facing product, we need to make sure that the Mozilla community can access, test, and develop around a solid product.

Strategy

Create a consistent and repeatable set of tests for qualifying and releasing the weekly builds and deployments for BrowserID. The focus needs to be on quality, but also on providing accurate feedback on successes and issues to Dev and PM so that the Production releases provide a useful product for the Mozilla and development communities.

Ad Hoc testing to define test paths on the RP/client and server side.
Create useful RP/client-side and server-side test cases that can be imported into Litmus for repeatable manual testing.
Create automated RP/client and server tests that can more accurately cover the full functionality of the UI, the API, and the main roles (PIA, RP, IP) and various flows.
Emphasis on security and privacy issues that might come up across OS, browsers, accounts, and emails on the various RPs/clients tested against the server.
Identifying and tracking issues in GitHub and Bugzilla.
Provide useful metrics to Dev and PM.

Scope of Testing

RP/Client-side testing will cover the following areas: basic functionality and UI, accounts and emails, interaction with the Server, security/privacy, and usability/compatibility across OS and browsers.
Server-side testing will cover the following areas: basic functionality, support for multiple RP/client sites, user security/privacy, communication with various RPs/clients, information handling and storage, security/privacy of information, information persistence across deployments, and logging.
Areas not covered
- Server and RP/Client automation
- Security: VEP, VES, VEC
- Flows/End-To-End
- Additional Stability Testing
- FireFox Add-On
- Internationalization/Localization
- Certificate length and session timeout/length
- Primary/secondary identifiers
You can find notes on these areas at the bottom of the Test Plan

General Test Information

Links and Documentation

Services QA BrowserID Site:
- BrowserID Releases: https://wiki.mozilla.org/QA/Browser_Technologies/Services/Releases/BrowserID

BrowserID Weekly Trains Testing/Process wiki:
- https://wiki.mozilla.org/QA/BrowserID/BrowserID_Weekly_Trains_Beta

OS/Platform/Browser Test Matrix
- This is still a work in progress, but can be found in Google docs here:
- https://docs.google.com/spreadsheet/ccc?key=0ArxosXOjCBacdHNrWFdBQmE3RU0xRFFwMUcxQWdRcXc&hl=en_US

Test Cases in Moztrap:
- https://moztrap.allizom.org/manage/cases/?filter-productversion=2
- and here: https://moztrap.mozilla.org/manage/cases/?filter-suite=82

Weekly Test Schedules

BrowserID testing runs on a weekly "train" or schedule:
- Wednesday afternoon: deployment to Production, Stage (QA)
- Thursdays/Fridays: open testing and experimentation by Dev, QA, and community
- Following week: Monday - Wednesday: QA testing and sign off of current deployment
NOTE: Dev deployments are now back in the hands of the Dev team, so they can control incremental builds and releases.
NOTE 2: OPs is working on a Test/CI environment that will be deployed nightly for Dev and QA testing purposes (automation, etc.).

Weekly Meetings

BrowserID Team Meeting: Mondays, 10:30am PST/PDT in Vidyo Rm: Identity
- Wiki: https://wiki.mozilla.org/Identity/WeeklyMeeting
BrowserID Bug Triage Meeting: Thursdays, 9:30am PST/PDT in Vidyo Rm: Identity
- https://github.com/mozilla/browserid/issues

Email and IRC

Post to this email List: https://lists.mozilla.org/listinfo/dev-identity
Or post to the Google Group:
- http://groups.google.com/group/mozilla.dev.identity/topics
- or https://groups.google.com/forum/?hl=en#!forum/mozilla.dev.identity
Post to IRC: #identity

Key players in IRC: #identity
- lloyd = Lloyd Hilael developer
- stomlinson = Shane Tomlinson developer
- benadida = Ben Adida developer
- thunder = Dan Mills PM
- gene = Gene Wood OPs
- jbonacci = James Bonacci QA
- jrgm = John Morrison QA
- ozten = Austin King developer

Bugs and Open Issues

Security, OPs, and Apps-related bugs in Bugzilla
- Classification: Client Software
- Component: Server: Identity
- Product: Mozilla Services
Deployment Requests in Bugzilla
- Classification: Client Software
- Product: Mozilla Services
- Component: Operations: Deployment Requests

BrowserID RP/Client and Server Bugs/Issues
- GitHub server: https://github.com/mozilla/browserid/issues
- GitHub RP/client: https://github.com/mozilla/123done/issues
- GitHub RP/client: https://github.com/lloyd/myfavoritebeer.org/issues

See the following site for more information on filing issues in GitHub:
- https://wiki.mozilla.org/QA/BrowserID/BrowserID_Weekly_Trains_Beta#Filing_Issues_Against_BrowserID

Quick Architectural Overview of Stage

The Stage environment is a set of physical Linux systems that run the various BrowserID services/processes:
- Webheads: These servers run the main browserid services and the verifier services.
- Secure Webheads: These servers run the dbwriter services.
- Keysigners: These servers run the keysigner services.
- Databases: These servers do all the MySQL work.
- Zeus: the front end of the Stage cluster, handles the load balancing of all the requests.

Specific RP/Client Features

TBD based on updated content and UI flows for http://123done.org, including session length for personal vs. shared computers.

RP/Client and Server Test Environments

Stage Environment:
- Server: https://login.anosrep.org/ (https://anosrep.org, was https://diresworb.org/)
- RP/Client 1: http://beta.myfavoritebeer.org/
- RP/Client 2: http://beta.123done.org/
- Primary 1: https://eyedee.me/

Dev Environment:
- Server: https://login.dev.anosrep.org (https://dev.anosrep.org, was https://dev.diresworb.org/)
- RP/Client 1: http://dev.myfavoritebeer.org/
- RP/Client 2: http://dev.123done.org/
- Primary 1: https://eyedee.me/

Production Environment:
- Server: https://login.persona.org (https://persona.org, was https://browserid.org/)
- RP/Client 1: http://www.myfavoritebeer.org/
- RP/Client 2: http://123done.org/
- Other working RPs/Clients: any Mozilla internal or public site using Persona/BrowserID
- Primary 1: https://eyedee.me/
- Primary 2: https://mockmyid.com/

Supported OS and Browsers

All information about supported platforms, operating systems, browsers, mobile devices is now kept in a Google doc spreadsheet:
https://docs.google.com/spreadsheet/ccc?key=0ArxosXOjCBacdHNrWFdBQmE3RU0xRFFwMUcxQWdRcXc&hl=en_US

Stage Logging

Log file locations on all services except Zeus:
- /var/log/browserid -> /var/browserid/log
Log file locations on Zeus:
- /var/log/zeus

webheads: /var/log/browserid
- browserid.log
- router.log
- verifier.log
- verifier-metrics.json
- verifier-compute.log
- static.log
- router-metrics.json

Note: router-metrics.json replaces browserid-metrics.json

secure webheads: /var/log/browserid/
- dbwriter.log

keysigners have /var/log/browserid/
- keysigner.log
- keysigner-compute.log

dbs: various db logs

zeus: various zeus logs

Note: the browserid-related logs currently are not being rotated or archived in Dev or Stage, but they are in Production.

Major Test Areas

Code Verification

Verify fixes/code changes by direct inspection (as required)

Server Source: https://github.com/mozilla/browserid
RP/Client source: https://github.com/lloyd/myfavoritebeer.org/
RP/Client source: https://github.com/mozilla/123done
Primary source: https://github.com/lloyd/eyedee.me/

Or, you can verify fixes by checking the commits listed in the Closed issues.

Note: This is only for cases where QA has no direct/easy way to verify the fix in using BrowserID.

Installation/Deployment Verification

Verify the deployment version, processes, and logging in Stage.

Stage Processes
- webheads:
  - node bin/browserid
  - node bin/verifier
  - node bin/router
  - node bin/static
  - /usr/bin/node /opt/browserid/lib/verifier/verifier-compute.js
  - /usr/bin/node /opt/browserid/lib/bcrypt-compute.js

- secure webheads:
  - node bin/dbwriter
  - /usr/bin/node /opt/browserid/lib/bcrypt-compute.js

- keysigners:
  - node bin/keysigner
  - /usr/bin/node /opt/browserid/lib/keysigner/keysigner-compute.js

- databases:
  - /bin/sh /usr/bin/mysqld_safe --datadir=/data/mysql --pid-file=/var/run/mysqld/mysqld.pid
  - /usr/sbin/mysqld --basedir=/usr --datadir=/data/mysql --user=mysql --log-error=/data/mysql/db1.iddb.scl2.stage.svc.mozilla.com.err --open-files-limit=8192 --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
  - perl /usr/bin/pt-heartbeat --create-table -D browserid -u root --update

- Note: This assumes access to the main Stage Admin server. If you do not have SSH access, then get QA staff or Dev to verify the processes.

Version Checks
- Production: https://login.persona.org/ver.txt
- Dev: https://login.dev.anosrep.org/ver.txt
- Stage: https://login.anosrep.org/ver.txt
- Note: Dev should always be running one version newer than Stage. Stage should be running one version newer than Production.

Verify Stage logging is active
- See Stage Logging section above.
- Note: This assumes access to the main Stage Admin server. If you do not have SSH access, then get QA staff or Dev to verify that the logs are active.

Local Installs

Verify at least one local install to Mac or Linux
Verify at least one local install with locale support to Mac or Linux
Run the unit tests on one local install (see next section)
See the following wiki for more information on basic/local installs:
- https://wiki.mozilla.org/QA/BrowserID/BrowserID_Basic_Install

Unit Testing

Dev has created three sets of unit tests
- Front-End Unit Tests - run in any browser
- Back-End Unit Tests - run from the BrowserID install directory
- "Headless" Front-End Unit Tests - run without a browser, from the BrowserID install directory

The QA goal is to run these unit tests in a local install and report results.
This testing, along with the Sanity/Acceptance/Smoke testing below, will be conditions for entering full Stage cycle testing.

See the Unit Test information here:
- https://wiki.mozilla.org/QA/BrowserID/BrowserID_Basic_Install#Running_Unit_Tests
Instructions are good for any installation...

Automation Testing

There are two main ways to run automation testing:
- Directly through the Selenium/Jenkins/Sauce Labs automation environment
- Locally using a stand-alone Selenium server and the built-in automation tests

Selenium/Jenkins/Sauce Labs automation - see the following links
- https://wiki.mozilla.org/QA/BrowserID/BrowserID_Basic_Install#Running_the_Automation_Tests
- https://wiki.mozilla.org/Identity/QA#BrowserID_Automation
  - Selenium BID dashboard: BrowserID dashboard: http://qa-selenium.mv.mozilla.com:8080/
  - Browserid-tests repo: https://github.com/mozilla/browserid/tree/dev/automation-tests
  - Dave Hunt's BIDPOM (BrowserID Page-Object Model) repo: https://github.com/davehunt/bidpom

Automation on the localhost - see the following links:
- https://wiki.mozilla.org/QA/BrowserID/BrowserID_Basic_Install#Running_the_Automation_Tests

Sanity/Acceptance/Smoke

Small, repeatable set of tests with known, good, expected results.
Manual and automated testing on client and server to pass a minimal level of acceptance without which, QA testing of BrowserID can not proceed.
See the Test Cases (listed in the "Links and Documentation" section above) for more information.

Server-side (https://login.anosrep.org/) MozTrap AUTO

Client-side (http://myfavoritebeer.org or http://123done.org using one desktop OS with one browser)
- Sign In using one RP/client to create a new account with one email
  - Verify correct email verification sent to correct account (email provider)
- Logout/Sign In/Logout using the same account from a different RP/client
- Sign In and add one BID email to the same account from a different RP/client
  - Verify correct email verification sent to correct account (email provider)
- Sign In and add one primary email to the same account
- Sign In and change the password on the same account
  - Verify correct email verification sent to correct account (email provider)
- Sign In to the same account and "sign out" by using the This is not me... link
- Delete/Remove the account from the server-side

Client-side (http://beta.myfavoritebeer.org or http://beta.123done.org using Android with stock browser)
- Sign In using one RP/client to create a new account with one email
  - Verify correct email verification sent to correct account (email provider)
- Logout/Sign In/Logout using the same account from a different RP/client
- Sign In and add one BID email to the same account from a different RP/client
  - Verify correct email verification sent to correct account (email provider)
- Sign In and add one primary email to the same account
- Sign In and change the password on the same account
  - Verify correct email verification sent to correct account (email provider)
- Sign In to the same account and "sign out" by using the This is not me... link
- Delete/Remove the account from the server-side

Bug Verification

Manual testing of bugs/issues resolved for this weekly cycle of testing.
Dev will list all resolved/closed issues in the deployment ticket for the weekly cycle of testing.
Test cases generated during this testing can be moved to an automation tool for bug regression (see below).
- Note in GitHub all the issues marked with the "needs testcase" label

Basic Functional

Manual and automated testing on the client and the server to verify basic functionality of BrowserID:

Site Access - Stage
- Verify use of HSTS (for https:// only sites)
- Verify that the Stage server site can be accessed through secured HTTPS: https://login.anosrep.org/
- Verify that the Stage server site can not be accessed using only HTTP, but instead gets redirected to the HTTPS site: http://login.anosrep.org/
- Verify that the Stage RP/client site can be accessed through standard HTTP: http://beta.myfavoritebeer.org/ or http://beta.123done.org
- Verify that the Stage RP/client site gets a certificate warning/error or is blocked when accessed through secured HTTPS: https://beta.myfavoritebeer.org/, http://beta.123done.org
- Verify similar for the following RPs/clients:
  - carrera.databits.net:9999/

Accounts and Email Verification
- Creating an account from the server site using Sign Up
- Creating an account inline (at first use of an RP/client Sign In) AUTO
- Email notifications for new accounts: verification email through email provider with proper email account listed, live verification link, etc. AUTO
  - Test with emails/accounts on various, popular email servers/services
  - See more details in the next bulleted list
- Creating multiple accounts with one or more emails
- Deleting one or more accounts (cancellation) from the server site

Email Notifications for server or RP/Client
- On the same OS
  - Email notification and verification using one browser AUTO
  - Email notification on one browser, verification on an another
- Across multiple OS
  - Email notification on one specific browser on one specific OS
  - Verification on the same browser on a different OS
- Other cases
  - Email notification on one specific browser on one specific OS
  - Verification on a different browser on a different OS
- Mail Servers
  - Check functionality when the user does not verify by email (skips, forgets)
  - Check functionality when the user can not verify by email (email provider is down or user can not access email account for some reason)
- BrowserID Server site
  - Check functionality when BrowserID server is unavailable (down or user is off the net)
  - Check functionality when BrowserID server is available but user has slow connection (like a public wifi)
- Stage server logs (if you have access)
  - Verify for each new email added, a verification string shows up in the browserid.log file
  - For example: {"level":"info","message":"\u001b[90mGET /add_email_address?token=p4FjEoHkwH4vNkUDPPFDrkS3uq58Faj01cGiZ1oGBY6MsyKt \u001b[32m200 \u001b[90m12ms\u001b[0m","timestamp":"2 Dec 13:45:54"}
  - It should look similar to the clickable verification link sent over email
  - For example: https://diresworb.org/add_email_address?token=p4FjEoHkwH4vNkUDPPFDrkS3uq58Faj01cGiZ1oGBY6MsyKt

Accounts and Emails
- Adding additional emails to an account AUTO
- Attempt to add an email (that may or may not be yours) from another account
- Deleting one or more emails (without actually deleting the account)
- Leaving/returning to sites (while signed in, after signing out)
- Browser restart after creation of account or access of an account
- Always logging out from sites vs. never logging out from sites (session timeouts)
- Shared access to same computer or profiles or accounts with different users
- Browser settings and preferences, esp. pop-ups, cookies, security, privacy
- Cancelling accounts
- Copy/Pasting emails (names) or passwords from other sources
- Auto-completion of emails and passwords
- Merging one or more accounts (that may or may not be yours)

Email and Password fields
- Email strings/types/limits
- Verify all legal combinations of characters for both "local name" and "domain name" parts of the email string
- Password strings/types/limits

More on Emails and Passwords
- Use of passwords (strict) vs. pass phrases
- Verify minimum/maximum sizes of emails and passwords (length)
- Password reset, password remember/restore
- Unique/unusual/edge case emails and passwords
- Email and Password character compatibility
- Valid vs. invalid email formats
- Valid vs. invalid password formats
- Different accounts using same email/password combos
- Try to setup a new account with a password and/or email already in use
- Copy/Pasting passwords from other sources
- Verify that passwords are never stored in LocalStore on the user machine
- Verify whether or not passwords are stored client-side (Stage RP)
- Verify whether or not passwords are stored on the server (Stage server)
- Verify proper formatting with very long emails and/or passwords.

Cross site activity
- Creating an account on one RP/client, verifying the account/email on another RP/client
- Adding an email on one RP/client while logged into another RP/client
- Deleting an email on one while logged into other RPs/clients
- Cancel account/delete email from the server while signed into an RP/client
- Deleting the whole account while logged into one or more RPs/clients
  - This must be done from the server
- Verify how account information on the server reflects the changes for each of these tests

Other Areas
- Verify that the user cannot log in with an email if he/she did not confirm the used email
- Log in simultaneously in two different browsers with the same email, then log out from one of the two browsers
- Log in with different emails for different clients in the same browser/different browser
- Log in with the same email for different clients in the same browser, then log out from one of the browsers
- Verify that the log in is kept when restoring a session after a browser crash (or after closing a browser with the session restore option selected (is this Firefox only?)

Support for Primaries

Verify basic Primary support through the use of one or more test Primary sites
Primary: https://eyedee.me/
Other sites: TBD

Links
Verify all links on the Primary, especially if they open a new page/tab or replace the current page/tab

Email and Password fields
- Email strings/types/limits
- Verify all legal combinations of characters for both "local name" and "domain name" parts of the email string
- Password strings/types/limits

Direct Primary usage
- Signing in from Primary
- Sign-in redirection to Primary from an RP/client
- Sign-out from Primary
- Sign-out from Primary while signed into an RP/client
- Sign-out from Primary while signed into the Server site

BrowserID tests (from Server or from RPs)
- Creating an account with Primary emails
- Creating an account with mixed emails (Primary/BID)
- Adding/Deleting a Primary email from a Primary account
- Adding/Deleting a BID email from a Primary account
- Adding/Deleting a Primary email from a BID account
- Adding/Deleting a BID email from a BID account
- Deleting an account with Primary emails
- Deleting an account with mixed emails (Primary/BID)
- Account Manager password changes on accounts with both primary and secondary emails

Network interruptions: latency or Primary site being down

Primary site UI flow
- General tests for navigating the site
- Include failures, cancellations, backing out

Local Installs: run the check_primary_support script on eyedee.me, other Primaries, and at least one non-primary

Regression: attempt Primary use with and without third-party cookie support set in the browser

Specific RP/Client Functionality

http://beta.myfavoritebeer.org
- Verify that this site allows for standard authentication, sign in, log out, password changes, and email additions to a new or current account.
- Verify different sign-in/verification paths for Primary emails vs. BID emails.
- Regression: attempt RP Sign In use with and without third-party cookie support set in the browser

http://123done.org
- Add some specific test information here...

UI

Manual and automated testing on the RP/client and the BID server to cover all aspects of the current UI.

Stage: Account creation UI in https://www.diresworb.org/
Stage: Account creation inline using http://beta.myfavoritebeer.org/ or http://123done.org
- Stage: Account creation inline using Primary support

Sign-in UI
Email field
Password field
"forgot your password?" link
"learn more" link
"This is not me" link
"Use a different email" link
"Terms of Service" and "Privacy" links off of RP sites

Account management: https://www.diresworb.org Account Manager page
Sign Up
Sign In
Edit button
Remove button
Password button
"cancel your account" link

UI for new user
- Email field
- Verify button
- Verification email
- Verification link
- Password
- Email selection - radio button vs. email string

Confirm your Email UI - verifying accurate "prove" link
- Confirm email verification from client-side and server-side, as defined
Confirm Email verification UI

RP/Client-side UI after sign-in (http://beta.myfavoritebeer.org, http://123done.org)
- There is a hello message of sorts: Yo, <NAME>!
- A dummy logo
- A logout link

RP/Client links:
BrowserID: https://diresworb.org
Source code: https://github.com/lloyd/myfavoritebeer.org/
Source code: https://github.com/mozilla/123done
Mozilla Labs: http://mozillalabs.com/

Server side UI:
BrowserID: https://diresworb.org/
How It Works: https://diresworb.org/about
Take The Tour: https://diresworb.org/about
Developers: https://github.com/mozilla/browserid/wiki/How-to-Use-BrowserID-on-Your-Site
Identity Team: http://identity.mozilla.com/
Mozilla Labs: http://mozillalabs.com/
Privacy: https://diresworb.org/privacy
TOS: https://diresworb.org/tos
Need Help: https://support.mozilla.com/en-US/kb/what-browserid-and-how-does-it-work
Sign In: https://diresworb.org/signin
Sign Up: https://diresworb.org/signup
Account Manager (once you are signed in)

UI to Add new emails ("Use a different email") while still logged into RP/client
UI to Change a password ("Forgot your password?") on the RP/client
UI to select Terms of Services, Privacy, Learn More on the RP/client
UI to Edit the account on the Server from the Account Manager
UI to Remove an email on the Server from the Account Manager
UI to Change the Password on the Server from the Account Manager
- Verify functionality with only BID emails
- Verify functionality with only Primary emails
- Verify functionality with a mix of email types
UI to Cancel an account (all emails) on the Server

General UI navigation to cover the following:
- Error screens and dialogs
- Email verification pop-ups and in browser (the fade/change to a server page)
- Closing pop-ups manually rather than with a Cancel, OK, Continue, or other button

Various UI scenarios and navigation not covered above...
- The affects of Stay logged in vs. always logging out

General ease of use
- Mouse/pointer: left (click), right/ctrl click, hover over links, etc.
- Keyboard: selection, tabbing, arrows, etc.
- Mobile: touchscreen, keyboard, etc.

Tool Tips, Messages, and Errors

Verify tool tips for all text fields and buttons (if supported).
Verify all error messages for various normal activities with BrowserID
Verify all error messages for various browser-level activities (browser reset, auto updates, reset of prefs, cookie deletion, etc.)
Verify all error messages for various system-level activities (network issues primarily)
Verify all error messages/pop-ups for disabled main/third-party cookies

Localization

Verify the UI flow with specific focus on localization, all text-based content
- Page content, layout, and strings
- Pop-up content, layout, and strings
- Email content, layout, and strings
- Messages, tool tips, errors
  - Errors: will need to create/force error situations in order to verify localization of error strings

List of supported languages
- From a local install, check the following files:
  - config/l10n-prod.json (for Production)
  - config/l10n-all.json (for Dev, local installs, and Stage)
  - l10n-prod.json should always be a subset of l10n-all.json
- For Stage to Prod deployments, check the very latest pull request for config/l10n-prod.json
  - Example: https://github.com/mozilla/browserid/pull/1563
- Note: Focus will always be on languages that are fully translated
REF: http://svn.mozilla.org/projects/l10n-misc/trunk/browserid/
REF: https://localize.mozilla.org/projects/browserid/

Setting locale in a browser
- Firefox: Preferences > Content > Languages
- Safari: Apple menu > System Preferences, and then click Language & Text.
  - Open Language & Text preferences
  - Drag your preferred language to the top of the Languages list.
  - If Safari is open, quit it, and then open it again.
  - NOTE: In the Language tab, you will need to click on "Edit List..." and add any special or test languages as a selectable language.
- Chrome: Tools > Preferences > Under the Hood > Web Content > Languages and Spell-Checker Settings
- Opera: Preferences > General > Language
  - Apparently can only pick through languages here: /Applications/Opera.app/Contents/Resources/
- IE8: Tools > Internet Options > General > Appearance > Languages > Language Reference > Add...
  - You can also create a User-Defined language by adding the language code directly
- IE9: Tools > Internet Options > General > Appearance > Languages > Language Reference > Add...
  - You can also create a User-Defined language by adding the language code directly
- iOS/Safari: TBD
  - Note: This will change the locale (at least temporarily) for all apps on iPhone and/or iPad

Quick verification of localized headers
- Here is a good site: http://www.ericgiguere.com/tools/http-header-viewer.html
  - Scroll down to the "Your Headers" section and look for "accept-language" row.
  - The ordered language list should show up there.
  - Example: eo,en-us;q=0.5

OS-specific changes, if any:
- Ex: Rebooting to clear cache, data, cookies, etc.
- Ex: text/font encodings
- Custom language entries for browsers

Accessibility - Desktop only

Verify minimal accessibility in the UI - keyboard only
- Creating a new BrowserID account
  - Email verification from mail application and from BrowserID
- Adding a new email to a current BrowserID account
  - Email verification from mail application and from BrowserID
- Changing/resetting a password - RP flow
  - Email verification from mail application and from BrowserID
- Changing a password - Account Manager
- Deleting an email from an account - Account Manager
- Deleting an account - Account Manager
- Other UI flows: This is not me, BrowserID links, etc.

Note: this testing may require changes at the OS-level (like Mac OS) or changes at the browser level to more fully support keyboard-only access to a site.

Bug Regression

Manual and automated testing of Verified/Closed bugs and issues from previous cycles of testing.
The list of bugs to regress are always listed from the ChangeLog link in the deployment ticket.
REF
- Server: https://github.com/mozilla/browserid/issues
- RPs/Clients: https://github.com/lloyd/myfavoritebeer.org/issues
- and https://github.com/mozilla/123done/issues

Ideally, this regression would be in the form of a quick automated test/script to check key fixes/changes/updates.
QA to work with Dev and PM to come up with a list of bugs/issues candidates to from the test cases for manual and automated testing.

Tagging Regressions
- If an old issue is found to exist in the code, then file a new issue and tag it with [regression] in the Title
- Give that regression issue a 5-star label

OS/Browser integration

Manual testing/verification of the required OS and browser configurations:

Verify access and use across browsers on same OS, different OS
Verify access and use across OS platforms, same browser, different browser
This includes account creation/update/deletion and email addition/deletion
OS/Browser-specific local storage verification
Browser preferences, esp for privacy/security
Browser synchronization - same platform, across platforms

Verify RP and main site access
- With and without third-party cookies enabled
- With and without all cookies enabled

Verify cached headers, status, content across browsers

Verify browser cookie information/inspection (per browser)
- Inspect the client and server browser cookie information for BrowserID specific information
- Privacy settings in browsers: start with 3rd-party cookies due to issues with eyedee.me
Verify door hanger information (per browser, desktop and mobile)
- Inspect/verify the doorhanger information and links (if any)

Also, QA needs to verify the clearing of all BrowserID data from a specific OS/browser/profile.
- See the following for more information:
- https://wiki.mozilla.org/QA/BrowserID/BrowserID_Weekly_Trains_Beta#Browser_and_Mobile_Support

On the FireFox browser, various tools can be used to help test/verify BrowserID:
- Tools > Web Developer > Web Console: can be used to track live GET and POST messages from the RP (client) site.
- Tools > Web Developer > Error Console: to track any errors, warnings, or messages from the RP (client) site.
- From the mobile version: TBD

On the Safari browser:
- Preferences > Advanced > "Show Develop menu in menu bar"
- Now, reference all tools through the Develop drop-down menu"
  - Error Console
- From the mobile version: Settings > Safari > Advanced > Debug Console > On

On the Google Chrome browser:
- Tool icon > Tools > Developer Tools
- View > Developer > Developer Tools

On the Opera browser:
- View > Developer Tools > Error Console

On the IE8/IE9 browsers:
- F12 or Developer Tools

On the Android Stock browser: TBD

BrowserID Logs/Messages

Verify that the main BrowserID logs contain meaningful information for inactive and active messaging:
- browserid.log
  - QA is most concerned about any warnings or errors that come up during testing, and any debug messages that do not belong (talk to Dev about list of current debug messages)
- verifier.log
  - QA should not see any debug, warn, or error messages in this log (verify with Dev)
- keysigner.log
  - QA should not see any debug, warn, or error messages in this log (verify with Dev)
- dbwriter.log
  - QA should not see any debug, warn, or error messages in this log (verify with Dev)
- All *-error.log files should be empty. If they are not, this could point to an issue requiring debug and Dev/OPs interaction.
- NOTE: archived logs (in gzip format) may need to be reviewed if testing is generating a lot of log activity or if issues need to be debugged over two or more log rotations.

Mobile Testing

Mobile-specific tests should be added to verify compatibility/stability/functionality of BrowserID across Android and iOS tablets and phones

Complete the Sanity/Acceptance section for phone, for tablet
Complete the Basic Functional and UI sections on phone, on tablet
- Panning, zooming, scrolling, tab access and flow between tabs
Complete the Specific RP/Client tests on phone, tablet
OS/Browser interaction using FireFox and stock browser on Android phones and tablets
OS/Browser interaction using Safari browser on iPhones and iPads
Compatibility - review actions and processes that are unique to mobile devices
Verify compatibility and functionality when working with Android/iOS Settings and with browser app-specific settings.
Most of the testing will be based on the final matrix located here:
- https://docs.google.com/spreadsheet/ccc?key=0ArxosXOjCBacdHNrWFdBQmE3RU0xRFFwMUcxQWdRcXc&hl=en_US

Network/Internet

Verify the impact of various ways to access the internet: timing, communication, messaging, outages, errors, traffic, etc.

Ethernet - personal vs. office, with and without VPN
Public WiFi
Private WiFi and other home setups
3g, 4g

Client-side LocalStorage

Most, if not all, of the BrowserID-specific information is now kept on the server.
But, LocalStorage should still be examined for any unexpected changes based on BrowserID use.
Work with Dev to come up with meaningful tests to verify BrowserID's use of, and impact on, LocalStorage.
Example issue: 688: IE8 blowing up trying to do "delete" on localStorage members.
All session data is stored in the cookie (per Ben Adida).

Server-side Testing

Verify some UI elements and actions only found on the server
- See UI section above.

Log inspection to verify accurate information
- See the Logging section above for more details

Load Testing and Monitoring
- See OPs section below for more information

API automation (if applicable/useful)

Unit Tests (work with Dev)

Headless frontend testing (work with Dev)

QA should look into headless (no browser/client) testing of server

Server/service scenarios - failover/recover, temp outage, traffic flow, etc.
- Zeus/load balancer issues
- webhead for browserid, verifier, router, and static
- swebhead for dbwriter
- keysigner for keysigner
- mysql/db

Load Testing

Load Testing in Stage is handled by QA using BrowserID code provided by Dev.
QA should monitor all load tests
QA should also schedule specific load tests to cover various requirements for server-side testing.
- Work with OPs to queue up specific load tests to check Server stability, various metrics
Monitor load tests with OPs dashboards - e.g., http://pencil.scl2.stage.svc.mozilla.com/dash/stage/bid_webhead (requires access to the MPT VPN)
Analyze text and visual data for functionality and peformance

REF:
- load_gen: https://github.com/mozilla/browserid/blob/dev/bin/load_gen

An example command line to run the load test:

    nohup bin/load_gen -s https://verifier.login.anosrep.org -o -m 250000 -u 1/250

-s  -- the URL to the browserid server
-o  -- don't request static resources
-m  -- maximum active users to simulate (ADU)
-u  -- use this range of #@loadtest.domain users (these users already exist in the 
       stage database. If running your own instance, do `CREATE_TEST_USERS=1000 
       BROWSERID_FAKE_VERIFICATION=1 NODE_ENV=test_mysql npm start`).

I typically run the above command piped into this:  

    | /usr/local/bin/tai64n | /usr/local/bin/tai64nlocal \
      >> /path/to/load_gen.log-`date +%Y%m%d-%H%M` 2>&1 &

which will timestamp the output and save it in a logfile, etc.

QA Sign-Off for Stage

Requirements and procedures:
- Complete all required testing for the current weekly train: resolved/closed issues, suggested areas for QA focus, specific features and areas of test coverage, automation, etc.
- Update the deployment ticket with a summary of the completed testing, plus the following: issues verified, issues opened, issues reopened, any testing marked incomplete, delayed, or removed from this release, etc.
- Add a sign-off to the deployment ticket if this weekly release passes QA
- Add additional comments (without a sign-off) if this weekly release fails QA

QA Testing for Production Pushes

This is a work in progress!...

Verify push to PHX1 (/etc/hosts hack)
- Standard sanity testing across platforms, OS, and browsers
Verify push to SCL2 (/etc/hosts hack)
- Standard sanity testing across platforms, OS, and browsers
Locale verification
- Based on final pull request for config/l10n-prod.json
- Quick sanity test of new/changed/removed locales
Check the verifier service before OPs undrains a datacenter during a push (for PHX1 and SCL2)
- REF: Bug 753728 - browserid.org/verify is on occasion throwing 500 errors (being caught on the load balancer)
- REF: Bug 753828 - monitor the state of zeus pools

QA Sign-Off for Production

TBD: Requirements and procedures:
Similar to sign-off for Stage, but applies to both data centers
- Add the Services Maintenance Window information to the deployment ticket.
- Complete all required testing for the push to Production.
- Update the deployment ticket with a summary of the completed testing.
- Add a sign-off to the deployment ticket if this Production push passes QA.
- Add additional comments (without a sign-off) if this Production push fails QA.

Back To ==> Mozilla BT Services