MozillaWiki

QA/BrowserID/TestPlan: Difference between revisions

Page Discussion

QA/BrowserID/TestPlan (view source)

Revision as of 01:29, 3 July 2013

24,134 bytes added ,  3 July 2013
→‎Sanity/Acceptance/Smoke
 
(284 intermediate revisions by 4 users not shown)
Line 1: Line 1:
Back To ==> [[QA/Browser_Technologies/Services | Mozilla BT Services]]
Back To ==> [[QA/Browser_Technologies/Services | Mozilla BT Services]]


== Overview ==
This test plan covers the general weekly testing that will happen against a built, unit-tested, and deployed BrowserID product in the Stage Test environment. The goal is to ensure a defined/consistent amount of quality and usability in the server-side and RP/client side portions of the BrowserID product. Since this is a public-facing product, we need to make sure that the Mozilla community can access, test, and develop around a solid product.


= Overview =
== Strategy ==
This test plan covers the general weekly testing that will happen against a built, unit-tested, and deployed BrowserID product in the Beta Test environment. The goal is to ensure a defined/consistent amount of quality and usability in the server-side and client side portions of the BrowserID product. Since this is a public-facing product, we need to make sure that the Mozilla community can access, test, and develop around a solid product.
 
= Strategy =
Create a consistent and repeatable set of tests for qualifying and releasing the weekly builds and deployments for BrowserID. The focus needs to be on quality, but also on providing accurate feedback on successes and issues to Dev and PM so that the Production releases provide a useful product for the Mozilla and development communities.
Create a consistent and repeatable set of tests for qualifying and releasing the weekly builds and deployments for BrowserID. The focus needs to be on quality, but also on providing accurate feedback on successes and issues to Dev and PM so that the Production releases provide a useful product for the Mozilla and development communities.


* Ad Hoc testing to define test paths on the client and server side.
* Ad Hoc testing to define test paths on the RP/client and server side.
* Create useful client-side and server-side test cases that can be imported into Litmus for repeatable manual testing.
* Create useful RP/client-side and server-side test cases that can be imported into Litmus for repeatable manual testing.
* Create automated client and server tests that can more accurately cover the full functionality of the UI, the API, and the main roles (PIA, RP, IP) and various flows.
* Create automated RP/client and server tests that can more accurately cover the full functionality of the UI, the API, and the main roles (PIA, RP, IP) and various flows.
* Emphasis on security and privacy issues that might come up across OS, browsers, accounts, and emails on the various clients tested against the server.
* Emphasis on security and privacy issues that might come up across OS, browsers, accounts, and emails on the various RPs/clients tested against the server.
* Identifying and tracking issues in GitHub and Bugzilla.
* Identifying and tracking issues in GitHub and Bugzilla.
* Provide useful metrics to Dev and PM.
* Provide useful metrics to Dev and PM.


= Scope of Testing =  
== Scope of Testing ==  
On the Client-side, testing will cover the basic functionality and UI, accounts and emails, interaction with the Server, security/privacy, and usability/compatibility across OS and browsers.
* RP/Client-side testing will cover the following areas: basic functionality and UI, accounts and emails, interaction with the Server, security/privacy, and usability/compatibility across OS and browsers.
* Server-side testing will cover the following areas: basic functionality, support for multiple RP/client sites, user security/privacy, communication with various RPs/clients, information handling and storage, security/privacy of information, information persistence across deployments, and logging.
* Areas not covered
** Server and RP/Client automation
** Security: VEP, VES, VEC
** Flows/End-To-End
** Additional Stability Testing
** FireFox Add-On
** Internationalization/Localization
** Certificate length and session timeout/length
** Primary/secondary identifiers
* You can find notes on these areas at the bottom of the Test Plan
 
== General Test Information ==
 
=== Links and Documentation ===
* Services QA BrowserID Site:
** BrowserID Releases: https://wiki.mozilla.org/QA/Browser_Technologies/Services/Releases/BrowserID
 
* BrowserID Weekly Trains Testing/Process wiki:
** https://wiki.mozilla.org/QA/BrowserID/BrowserID_Weekly_Trains_Beta
 
* OS/Platform/Browser Test Matrix
** This is still a work in progress, but can be found in Google docs here:
** https://docs.google.com/spreadsheet/ccc?key=0ArxosXOjCBacdHNrWFdBQmE3RU0xRFFwMUcxQWdRcXc&hl=en_US
 
* Test Cases in Moztrap:
** https://moztrap.allizom.org/manage/cases/?filter-productversion=2
** and here: https://moztrap.mozilla.org/manage/cases/?filter-suite=82
 
=== Weekly Test Schedules ===
* BrowserID testing runs on a weekly "train" or schedule:
** Wednesday afternoon: deployment to Production, Stage (QA)
** Thursdays/Fridays: open testing and experimentation by Dev, QA, and community
** Following week: Monday - Wednesday: QA testing and sign off of current deployment
* NOTE: Dev deployments are now back in the hands of the Dev team, so they can control incremental builds and releases.
* NOTE 2: OPs is working on a Test/CI environment that will be deployed nightly for Dev and QA testing purposes (automation, etc.).
 
=== Weekly Meetings ===
* BrowserID Team Meeting: Mondays, 10:30am PST/PDT in Vidyo Rm: Identity
** Wiki: https://wiki.mozilla.org/Identity/WeeklyMeeting
* BrowserID Bug Triage Meeting: Thursdays, 9:30am PST/PDT in Vidyo Rm: Identity
** https://github.com/mozilla/browserid/issues
 
=== Email and IRC ===
* Post to this email List: https://lists.mozilla.org/listinfo/dev-identity
* Or post to the Google Group:
** http://groups.google.com/group/mozilla.dev.identity/topics
** or https://groups.google.com/forum/?hl=en#!forum/mozilla.dev.identity
* Post to IRC: #identity
 
* Key players in IRC: #identity
** lloyd = Lloyd Hilael developer
** stomlinson = Shane Tomlinson developer
** benadida = Ben Adida developer
** thunder = Dan Mills PM
** gene = Gene Wood OPs
** jbonacci = James Bonacci QA
** jrgm = John Morrison QA
** ozten = Austin King developer
 
== Bugs and Open Issues ==
* Security, OPs, and Apps-related bugs in Bugzilla
** Classification: Client Software
** Component: Server: Identity
** Product: Mozilla Services
* Deployment Requests in Bugzilla
** Classification: Client Software
** Product: Mozilla Services
** Component: Operations: Deployment Requests


On the Server-side, testing will start with basic functionality, support for multiple client sites, user security/privacy, communication with various clients, then move on to information handling and storage, security/privacy of information, information persistence across deployments, logging.
* BrowserID RP/Client and Server Bugs/Issues
** GitHub server: https://github.com/mozilla/browserid/issues
** GitHub RP/client: https://github.com/mozilla/123done/issues
** GitHub RP/client: https://github.com/lloyd/myfavoritebeer.org/issues


= General Test Information =
* See the following site for more information on filing issues in GitHub:
** https://wiki.mozilla.org/QA/BrowserID/BrowserID_Weekly_Trains_Beta#Filing_Issues_Against_BrowserID


== Links and Documentation ==  
== Quick Architectural Overview of Stage ==
* Browser ID Sites:
* The Stage environment is a set of physical Linux systems that run the various BrowserID services/processes:
** Main site: https://wiki.mozilla.org/Identity
** Webheads: These servers run the main browserid services and the verifier services.
** Production server: https://browserid.org
** Secure Webheads: These servers run the dbwriter services.
** BrowserID Introduction and Demo: https://browserid.org/users
** Keysigners: These servers run the keysigner services.
** BrowserID How-To for developers: https://browserid.org/developers.html
** Databases: These servers do all the MySQL work.
** Zeus: the front end of the Stage cluster, handles the load balancing of all the requests.


* Services QA Sites:
== Specific RP/Client Features ==
** Mozilla QA main site: https://wiki.mozilla.org/QA
* TBD based on updated content and UI flows for http://123done.org, including session length for personal vs. shared computers.
** Mozilla QA main site: http://quality.mozilla.org/
** Browser Technologies QA: https://wiki.mozilla.org/QA/Browser_Technologies
** Services QA: https://wiki.mozilla.org/QA/Browser_Technologies/Services


== Weekly Test Schedules ==
== RP/Client and Server Test Environments ==  
* Thursdays: deployment to Production, Beta (QA), and Dev
* Stage Environment:
* Thursdays/Fridays: open testing and experimentation by Dev, QA, and community
** Server: https://login.anosrep.org/ (https://anosrep.org, was https://diresworb.org/)
* Following week:
** RP/Client 1: http://beta.myfavoritebeer.org/
** Monday - Wednesday: QA testing and sign off of current deployment
** RP/Client 2: http://beta.123done.org/
** Primary 1: https://eyedee.me/


== Weekly Meetings ==
* Dev Environment:
Weekly meeting notes are kept on the following EtherPad site:
** Server: https://login.dev.anosrep.org (https://dev.anosrep.org, was https://dev.diresworb.org/)
* http://etherpad.mozilla.com:9000/identity-weekly-meeting
** RP/Client 1: http://dev.myfavoritebeer.org/
** RP/Client 2: http://dev.123done.org/
** Primary 1: https://eyedee.me/


== Posting Feedback ==
* Production Environment:
* Email List: https://lists.mozilla.org/listinfo/dev-identity
** Server: https://login.persona.org (https://persona.org, was https://browserid.org/)
* Blog: http://identity.mozilla.com/
** RP/Client 1: http://www.myfavoritebeer.org/
* Google Group: http://groups.google.com/group/mozilla.dev.identity/topics
** RP/Client 2: http://123done.org/
* or https://groups.google.com/forum/?hl=en#!forum/mozilla.dev.identity
** Other working RPs/Clients: any Mozilla internal or public site using Persona/BrowserID
* IRC: #identity
** Primary 1: https://eyedee.me/
** Primary 2: https://mockmyid.com/


= Bugs and Open Issues =
== Supported OS and Browsers ==
== Ops Bugs in Bugzilla ==
* All information about supported platforms, operating systems, browsers, mobile devices is now kept in a Google doc spreadsheet:
* Classification: Other
* https://docs.google.com/spreadsheet/ccc?key=0ArxosXOjCBacdHNrWFdBQmE3RU0xRFFwMUcxQWdRcXc&hl=en_US
* Product: mozilla.org
* Component: Server Operations: Labs


== BrowserID Client/Server Bugs/Issues ==
== Stage Logging ==
* Add/update issues: https://github.com/mozilla/browserid/issues
* Log file locations on all services except Zeus:
** /var/log/browserid -> /var/browserid/log
* Log file locations on Zeus:
** /var/log/zeus


= Client/Server Test Environment =
* webheads: /var/log/browserid
== Beta test environment ==
** browserid.log
* Server-side: https://diresworb.org/
** router.log
* Client-side: http://beta.myfavoritebeer.org/
** verifier.log
* DNS/Load Balancer (no permissions): browserid-stage1.vm1.labs.sjc1.mozilla.com
** verifier-metrics.json
** verifier-compute.log
** static.log
** router-metrics.json
Note: router-metrics.json replaces browserid-metrics.json
* secure webheads: /var/log/browserid/
** dbwriter.log


== Prod test environment ==
* keysigners have /var/log/browserid/
* Server-Side (PIA/IP)
** keysigner.log
** Acts as the Implementation Provider (IP) until browsers have native support.
** keysigner-compute.log
** https://browserid.org
* Server Access (Localhost examples):
** https://browserid.org/verify: http://127.0.0.1:10000
** http://rp.eyedee.me: http://127.0.0.1:10001
** https://browserid.org: http://127.0.0.1:10002
** Are there any others?
* Client-Side (RP)
** Acts as the Relying Party (RP) until there are more public clients to test.
** www.myfavoritebeer.org


= Supported OS and Browsers =
* dbs: various db logs
== Operating Systems ==
* PC: Win 7, WinXP, Vista (in priority order)
* Mac 10.6, 10.7, 10.5
* Linux: Fedora 14/15, EL5/6, Ubuntu
* Mobile: Android, iOS


== Browsers ==
* zeus: various zeus logs
* Minimal support would have to include a native browser plus Firefox
** Windows: IE 8/9, FF 4/5, Google Chrome latest
** Mac: Safari latest, FF 4/5, Google Chrome latest
** Linux: FF 3/4/5, Google Chrome latest


= Logging =
* Note: the browserid-related logs currently are not being rotated or archived in Dev or Stage, but they are in Production.
There are four primary logs that should be monitored during testing: two for BrowserID and two for nginx:
* Nginx logs: /var/log/nginx/
** access.log
** error.log
** archived logs
* BrowserID (node) logs
** <install path>/var_browserid/server.log
** <install path>/var_verifier/server.log


= Major Test Areas =  
== Major Test Areas ==  
== Code Verification ==
=== Code Verification ===
Verify non-QA-testable fixes/code changes by direct inspection (as required)
Verify fixes/code changes by direct inspection (as required)
* Server Source: https://github.com/mozilla/browserid
* Server Source: https://github.com/mozilla/browserid
* Client Source: https://github.com/lloyd/myfavoritebeer.org/
* RP/Client source: https://github.com/lloyd/myfavoritebeer.org/
* RP/Client source: https://github.com/mozilla/123done
* Primary source: https://github.com/lloyd/eyedee.me/
 
* Or, you can verify fixes by checking the commits listed in the Closed issues.
 
* Note: This is only for cases where QA has no direct/easy way to verify the fix in using BrowserID.
 
=== Installation/Deployment Verification ===
Verify the deployment version, processes, and logging in Stage.
 
* Stage Processes
** webheads:
*** node bin/browserid
*** node bin/verifier
*** node bin/router
*** node bin/static
*** /usr/bin/node /opt/browserid/lib/verifier/verifier-compute.js
*** /usr/bin/node /opt/browserid/lib/bcrypt-compute.js
 
** secure webheads:
*** node bin/dbwriter
*** /usr/bin/node /opt/browserid/lib/bcrypt-compute.js
 
** keysigners:
*** node bin/keysigner
*** /usr/bin/node /opt/browserid/lib/keysigner/keysigner-compute.js
 
** databases:
*** /bin/sh /usr/bin/mysqld_safe --datadir=/data/mysql --pid-file=/var/run/mysqld/mysqld.pid
*** /usr/sbin/mysqld --basedir=/usr --datadir=/data/mysql --user=mysql --log-error=/data/mysql/db1.iddb.scl2.stage.svc.mozilla.com.err --open-files-limit=8192 --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
*** perl /usr/bin/pt-heartbeat --create-table -D browserid -u root --update
 
** Note: This assumes access to the main Stage Admin server. If you do not have SSH access, then get QA staff or Dev to verify the processes.
 
* Version Checks
** Production: https://login.persona.org/ver.txt
** Dev: https://login.dev.anosrep.org/ver.txt
** Stage: https://login.anosrep.org/ver.txt
** Note: Dev should always be running one version newer than Stage. Stage should be running one version newer than Production.
 
* Verify Stage logging is active
** See Stage Logging section above.
** Note: This assumes access to the main Stage Admin server. If you do not have SSH access, then get QA staff or Dev to verify that the logs are active.
 
=== Local Installs ===
* Verify at least one local install to Mac or Linux
* Verify at least one local install with locale support to Mac or Linux
* Run the unit tests on one local install (see next section)
* See the following wiki for more information on basic/local installs:
** https://wiki.mozilla.org/QA/BrowserID/BrowserID_Basic_Install
 
=== Unit Testing ===
* Dev has created three sets of unit tests
** Front-End Unit Tests - run in any browser
** Back-End Unit Tests - run from the BrowserID install directory
** "Headless" Front-End Unit Tests - run without a browser, from the BrowserID install directory
 
* The QA goal is to run these unit tests in a local install and report results.
* This testing, along with the Sanity/Acceptance/Smoke testing below, will be conditions for entering full Stage cycle testing.
 
* See the Unit Test information here:
** https://wiki.mozilla.org/QA/BrowserID/BrowserID_Basic_Install#Running_Unit_Tests
* Instructions are good for any installation...
 
=== Automation Testing ===
* There are two main ways to run automation testing:
** Directly through the Selenium/Jenkins/Sauce Labs automation environment
** Locally using a stand-alone Selenium server and the built-in automation tests
 
* Selenium/Jenkins/Sauce Labs automation - see the following links
** https://wiki.mozilla.org/QA/BrowserID/BrowserID_Basic_Install#Running_the_Automation_Tests
** https://wiki.mozilla.org/Identity/QA#BrowserID_Automation
*** Selenium BID dashboard: BrowserID dashboard: http://qa-selenium.mv.mozilla.com:8080/
*** Browserid-tests repo: https://github.com/mozilla/browserid/tree/dev/automation-tests
*** Dave Hunt's BIDPOM (BrowserID Page-Object Model) repo: https://github.com/davehunt/bidpom
 
* Automation on the localhost - see the following links:
** https://wiki.mozilla.org/QA/BrowserID/BrowserID_Basic_Install#Running_the_Automation_Tests
 
=== Sanity/Acceptance/Smoke ===
* Small, repeatable set of tests with known, good, expected results.
* Manual and automated testing on client and server to pass a minimal level of acceptance without which, QA testing of BrowserID can not proceed.
* See the Test Cases (listed in the "Links and Documentation" section above) for more information.
 
* Server-side (https://login.anosrep.org/) [https://moztrap.mozilla.org/manage/cases/?filter-productversion=2&filter-suite=82 MozTrap] [https://github.com/mozilla/browserid/blob/dev/automation-tests/persona_server/tests/test_manage_account.py AUTO]
** [https://moztrap.mozilla.org/manage/case/989/ Sign Up with a new account and one email]
** [https://moztrap.mozilla.org/manage/case/990/ Sign In - Sign Out with a current account of one email]
** [https://moztrap.mozilla.org/manage/case/992/ Change the password on a current account with one email]
** [https://moztrap.mozilla.org/manage/case/993/ Delete/Remove a current account with one email]
 
* Client-side (http://myfavoritebeer.org or http://123done.org using one desktop OS with one browser)
** Sign In using one RP/client to create a new account with one email
*** Verify correct email verification sent to correct account (email provider)
** Logout/Sign In/Logout using the same account from a different RP/client
** Sign In and add one BID email to the same account from a different RP/client
*** Verify correct email verification sent to correct account (email provider)
** Sign In and add one primary email to the same account
** Sign In and change the password on the same account
*** Verify correct email verification sent to correct account (email provider)
** Sign In to the same account and "sign out" by using the This is not me... link
** Delete/Remove the account from the server-side
 
* Client-side (http://beta.myfavoritebeer.org or http://beta.123done.org using Android with stock browser)
** Sign In using one RP/client to create a new account with one email
*** Verify correct email verification sent to correct account (email provider)
** Logout/Sign In/Logout using the same account from a different RP/client
** Sign In and add one BID email to the same account from a different RP/client
*** Verify correct email verification sent to correct account (email provider)
** Sign In and add one primary email to the same account
** Sign In and change the password on the same account
*** Verify correct email verification sent to correct account (email provider)
** Sign In to the same account and "sign out" by using the This is not me... link
** Delete/Remove the account from the server-side
 
=== Bug Verification ===
* Manual testing of bugs/issues resolved for this weekly cycle of testing.
* Dev will list all resolved/closed issues in the deployment ticket for the weekly cycle of testing.
* Test cases generated during this testing can be moved to an automation tool for bug regression (see below).
** Note in GitHub all the issues marked with the "needs testcase" label
 
=== Basic Functional ===
Manual and automated testing on the client and the server to verify basic functionality of BrowserID:
* Site Access - Stage
** Verify use of HSTS (for https:// only sites)
** Verify that the Stage server site can be accessed through secured HTTPS: https://login.anosrep.org/
** Verify that the Stage server site can not be accessed using only HTTP, but instead gets redirected to the HTTPS site: http://login.anosrep.org/
** Verify that the Stage RP/client site can be accessed through standard HTTP: http://beta.myfavoritebeer.org/ or http://beta.123done.org
** Verify that the Stage RP/client site gets a certificate warning/error or is blocked when accessed through secured HTTPS: https://beta.myfavoritebeer.org/, http://beta.123done.org
** Verify similar for the following RPs/clients:
*** carrera.databits.net:9999/
 
* Accounts and Email Verification
** Creating an account from the server site using Sign Up
** Creating an account inline (at first use of an RP/client Sign In) [https://github.com/mozilla/BrowserID-Tests/blob/master/123done/tests/test_new_user.py AUTO]
** Email notifications for new accounts: verification email through email provider with proper email account listed, live verification link, etc. [https://github.com/mozilla/BrowserID-Tests/blob/master/123done/tests/test_new_user.py AUTO]
*** Test with emails/accounts on various, popular email servers/services
*** See more details in the next bulleted list
** Creating multiple accounts with one or more emails
** Deleting one or more accounts (cancellation) from the server site
 
* Email Notifications for server or RP/Client
** On the same OS
*** Email notification and verification using one browser [https://github.com/mozilla/BrowserID-Tests/blob/master/123done/tests/test_new_user.py AUTO]
*** Email notification on one browser, verification on an another
** Across multiple OS
*** Email notification on one specific browser on one specific OS
*** Verification on the same browser on a different OS
** Other cases
*** Email notification on one specific browser on one specific OS
*** Verification on a different browser on a different OS
** Mail Servers
*** Check functionality when the user does not verify by email (skips, forgets)
*** Check functionality when the user can not verify by email (email provider is down or user can not access email account for some reason)
** BrowserID Server site
*** Check functionality when BrowserID server is unavailable (down or user is off the net)
*** Check functionality when BrowserID server is available but user has slow connection (like a public wifi)
** Stage server logs (if you have access)
*** Verify for each new email added, a verification string shows up in the browserid.log file
*** For example: {"level":"info","message":"\u001b[90mGET /add_email_address?token=p4FjEoHkwH4vNkUDPPFDrkS3uq58Faj01cGiZ1oGBY6MsyKt \u001b[32m200 \u001b[90m12ms\u001b[0m","timestamp":"2 Dec 13:45:54"}
*** It should look similar to the clickable verification link sent over email
*** For example: https://diresworb.org/add_email_address?token=p4FjEoHkwH4vNkUDPPFDrkS3uq58Faj01cGiZ1oGBY6MsyKt
 
* Accounts and Emails
** Adding additional emails to an account [https://github.com/mozilla/BrowserID-Tests/blob/master/123done/tests/test_add_another_email.py AUTO]
** Attempt to add an email (that may or may not be yours) from another account
** Deleting one or more emails (without actually deleting the account)
** Leaving/returning to sites (while signed in, after signing out)
** Browser restart after creation of account or access of an account
** Always logging out from sites vs. never logging out from sites (session timeouts)
** Shared access to same computer or profiles or accounts with different users
** Browser settings and preferences, esp. pop-ups, cookies, security, privacy
** Cancelling accounts
** Copy/Pasting emails (names) or passwords from other sources
** Auto-completion of emails and passwords
** Merging one or more accounts (that may or may not be yours)
 
* Email and Password fields
** Email strings/types/limits
** Verify all legal combinations of characters for both "local name" and "domain name" parts of the email string
** Password strings/types/limits
 
* More on Emails and Passwords
** Use of passwords (strict) vs. pass phrases
** Verify minimum/maximum sizes of emails and passwords (length)
** Password reset, password remember/restore
** Unique/unusual/edge case emails and passwords
** Email and Password character compatibility
** Valid vs. invalid email formats
** Valid vs. invalid password formats
** Different accounts using same email/password combos
** Try to setup a new account with a password and/or email already in use
** Copy/Pasting passwords from other sources
** Verify that passwords are never stored in LocalStore on the user machine
** Verify whether or not passwords are stored client-side (Stage RP)
** Verify whether or not passwords are stored on the server (Stage server)
** Verify proper formatting with very long emails and/or passwords.
 
* Cross site activity
** Creating an account on one RP/client, verifying the account/email on another RP/client
** Adding an email on one RP/client while logged into another RP/client
** Deleting an email on one while logged into other RPs/clients
** Cancel account/delete email from the server while signed into an RP/client
** Deleting the whole account while logged into one or more RPs/clients
*** This must be done from the server
** Verify how account information on the server reflects the changes for each of these tests
 
* Other Areas
** Verify that the user cannot log in with an email if he/she did not confirm the used email
** Log in simultaneously in two different browsers with the same email, then log out from one of the two browsers
** Log in with different emails for different clients in the same browser/different browser
** Log in with the same email for different clients in the same browser, then log out from one of the browsers
** Verify that the log in is kept when restoring a session after a browser crash (or after closing a browser with the session restore option selected (is this Firefox only?)
 
=== Support for Primaries ===
* Verify basic Primary support through the use of one or more test Primary sites
* Primary: https://eyedee.me/
* Other sites: TBD
 
* Links
* Verify all links on the Primary, especially if they open a new page/tab or replace the current page/tab
 
* Email and Password fields
** Email strings/types/limits
** Verify all legal combinations of characters for both "local name" and "domain name" parts of the email string
** Password strings/types/limits
 
* Direct Primary usage
** Signing in from Primary
** Sign-in redirection to Primary from an RP/client
** Sign-out from Primary
** Sign-out from Primary while signed into an RP/client
** Sign-out from Primary while signed into the Server site
 
* BrowserID tests (from Server or from RPs)
** Creating an account with Primary emails
** Creating an account with mixed emails (Primary/BID)
** Adding/Deleting a Primary email from a Primary account
** Adding/Deleting a BID email from a Primary account
** Adding/Deleting a Primary email from a BID account
** Adding/Deleting a BID email from a BID account
** Deleting an account with Primary emails
** Deleting an account with mixed emails (Primary/BID)
** Account Manager password changes on accounts with both primary and secondary emails
 
* Network interruptions: latency or Primary site being down


== Installation/Deployment Verification ==
* Primary site UI flow
QA needs access to Beta test environment in order to verify the deployment to server side (PIA/IP) and client side (RP). It would be helpful for QA to verify the weekly deployment to the server, along with any client-side changes (JavaScript library, JavaScript API). For early trains, this should be covered by information given in the BrowserID deployment ticket (Bugzilla). This would amount to a delta from the previous train that QA can record as part of their testing.
** General tests for navigating the site
** Include failures, cancellations, backing out


For subsequent trains, the Beta test environment or a QA environment (deployed with the same build) could be used to verify server deployments and client-side changes.
* Local Installs: run the check_primary_support script on eyedee.me, other Primaries, and at least one non-primary


== Sanity/Acceptance/Smoke ==
* Regression: attempt Primary use with and without third-party cookie support set in the browser
Small, repeatable set of tests with known, good, expected results. Manual and automated testing on client-side to pass a minimal level of acceptance without which, QA testing of BrowserID can not proceed. This will probably be a very small subset of the basic functional tests or some automated smoke test: TBD


== Bug Verification ==
=== Specific RP/Client Functionality ===
Manual testing of bugs/issues resolved for this weekly cycle of testing.
* http://beta.myfavoritebeer.org
Test cases generated during this testing can be moved to an automation tool for bug regression (see below).
** Verify that this site allows for standard authentication, sign in, log out, password changes, and email additions to a new or current account.
** Verify different sign-in/verification paths for Primary emails vs. BID emails.
** Regression: attempt RP Sign In use with and without third-party cookie support set in the browser


== Basic Functional ==
* http://123done.org
Manual and automated testing on the client and the server to verify basic functionality of BrowserID:  
** Add some specific test information here...
* Creating an account first
* Creating an account inline (at first use)
* Email verification for new accounts and emails
** Would be useful to have email accounts on various, popular email servers/services
** Need a method to verify this by direct inspection in server: comparing token in email link to information on server (or wherever this is temporarily stored)
* Creating multiple accounts
* Deleting one or more accounts (cancellation)
* Adding additional emails
* Deleting one or more emails (maintaining the account though)
* Forgotten account information - mail, passwords
* Leaving/returning to sites
* Browser restart after creation of identity
* Always logging out from sites vs. never logging out from sites
* Email and Password character compatibility
* Valid vs. invalid email formats
* Different accounts using same email/password combos
* Shared access to same computer or profiles or accounts
* Browser settings and preferences, esp. pop-ups, security, privacy
* FireFox about:config, similar, if any, for other browsers


=== UI ===
Manual and automated testing on the RP/client and the BID server to cover all aspects of the current UI.
* Stage: Account creation UI in https://www.diresworb.org/
* Stage: Account creation inline using http://beta.myfavoritebeer.org/ or http://123done.org
** Stage: Account creation inline using Primary support


== UI ==
Manual and automated testing on the client and the server to cover all aspects of the current UI:
(yes, this is a bit of an overkill for a Test Plan...)
* Account creation UI in https://www.browserid.org/
* Account creation inline using http://myfavoritebeer.org/ or other client
* Sign-in UI
* Sign-in UI
* Add a new email address
* Email field
* This is not me
* Password field
* Stay logged in vs. always log out
* "forgot your password?" link
* Account management: https://www.browserid.org/manage
* "learn more" link
* Account management through https://www.browserid.org, "my account" link
* "This is not me" link
* Account cancellation from https://www.browserid.org/manage
* "Use a different email" link
* HTML pop-ups: https://browserid.org/dialog/dialog.html
* "Terms of Service" and "Privacy" links off of RP sites
** Testing the various fields: email, password
 
** I forgot my password link: https://browserid.org/dialog/dialog.html#
* Account management: https://www.diresworb.org Account Manager page
** Don't have a BrowserID yet? link: https://browserid.org/dialog/dialog.html#
* Sign Up
*** Email field
* Sign In
*** Password field
* Edit button
*** Verify field
* Remove button
*** Go Back button
* Password button
*** Cancel button
* "cancel your account" link
*** Continue button (grayed out be default)
 
* Link to Mozilla Labs: http://mozillalabs.org/
* UI for new user
* terms: https://browserid.org/tos
** Email field
* privacy policy: https://browserid.org/privacy
** Verify button
* learn more: https://browserid.org/users
** Verification email
** Verification link
** Password
** Email selection - radio button vs. email string
 
* Confirm your Email UI - verifying accurate "prove" link
* Confirm your Email UI - verifying accurate "prove" link
** Confirm email verification from client-side and server-side, as defined
** Confirm email verification from client-side and server-side, as defined
* Confirm Email verification UI
* Confirm Email verification UI
** Pop-up changes to: "Your email has been confirmed!", with Cancel and Continue buttons.
 
* Client-side UI after sign-in
* RP/Client-side UI after sign-in (http://beta.myfavoritebeer.org, http://123done.org)
** There is a hello message of sorts: Yo, jabjab101@mailinator.com!
** There is a hello message of sorts: Yo, <NAME>!
** A dummy logo
** A dummy logo
** A logout link: http://myfavoritebeer.org/
** A logout link
* Add new emails while still logged into client
 
* Logout links on server side (if they exist)
* RP/Client links:
* Various UI scenarios and walkthroughs not covered above...
* BrowserID: https://diresworb.org
* Source code: https://github.com/lloyd/myfavoritebeer.org/
* Source code: https://github.com/mozilla/123done
* Mozilla Labs: http://mozillalabs.com/
 
* Server side UI:
* BrowserID: https://diresworb.org/
* How It Works: https://diresworb.org/about
* Take The Tour: https://diresworb.org/about
* Developers: https://github.com/mozilla/browserid/wiki/How-to-Use-BrowserID-on-Your-Site
* Identity Team: http://identity.mozilla.com/
* Mozilla Labs: http://mozillalabs.com/
* Privacy: https://diresworb.org/privacy
* TOS: https://diresworb.org/tos
* Need Help: https://support.mozilla.com/en-US/kb/what-browserid-and-how-does-it-work
* Sign In: https://diresworb.org/signin
* Sign Up: https://diresworb.org/signup
* Account Manager (once you are signed in)
 
* UI to Add new emails ("Use a different email") while still logged into RP/client
* UI to Change a password ("Forgot your password?") on the RP/client
* UI to select Terms of Services, Privacy, Learn More on the RP/client
* UI to Edit the account on the Server from the Account Manager
* UI to Remove an email on the Server from the Account Manager
* UI to Change the Password on the Server from the Account Manager
** Verify functionality with only BID emails
** Verify functionality with only Primary emails
** Verify functionality with a mix of email types
* UI to Cancel an account (all emails) on the Server
 
* General UI navigation to cover the following:
** Error screens and dialogs
** Email verification pop-ups and in browser (the fade/change to a server page)
** Closing pop-ups manually rather than with a Cancel, OK, Continue, or other button
 
* Various UI scenarios and navigation not covered above...
** The affects of Stay logged in vs. always logging out
 
* General ease of use
* General ease of use
** Mouse/pointer: left (click), right/ctrl click, hover over links, etc.
** Keyboard: selection, tabbing, arrows, etc.
** Mobile: touchscreen, keyboard, etc.
=== Tool Tips, Messages, and Errors ===
* Verify tool tips for all text fields and buttons (if supported).
* Verify all error messages for various normal activities with BrowserID
* Verify all error messages for various browser-level activities (browser reset, auto updates, reset of prefs, cookie deletion, etc.)
* Verify all error messages for various system-level activities (network issues primarily)
* Verify all error messages/pop-ups for disabled main/third-party cookies
=== Localization ===
* Verify the UI flow with specific focus on localization, all text-based content
** Page content, layout, and strings
** Pop-up content, layout, and strings
** Email content, layout, and strings
** Messages, tool tips, errors
*** Errors: will need to create/force error situations in order to verify localization of error strings
* List of supported languages
** From a local install, check the following files:
*** config/l10n-prod.json (for Production)
*** config/l10n-all.json (for Dev, local installs, and Stage)
*** l10n-prod.json should always be a subset of l10n-all.json
** For Stage to Prod deployments, check the very latest pull request for config/l10n-prod.json
*** Example: https://github.com/mozilla/browserid/pull/1563
** Note: Focus will always be on languages that are fully translated
* REF: http://svn.mozilla.org/projects/l10n-misc/trunk/browserid/
* REF: https://localize.mozilla.org/projects/browserid/
* Setting locale in a browser
** Firefox: Preferences > Content > Languages
** Safari: Apple menu > System Preferences, and then click Language & Text.
*** Open Language & Text preferences
*** Drag your preferred language to the top of the Languages list.
*** If Safari is open, quit it, and then open it again.
*** NOTE: In the Language tab, you will need to click on "Edit List..." and add any special or test languages as a selectable language.
** Chrome: Tools > Preferences > Under the Hood > Web Content > Languages and Spell-Checker Settings
** Opera: Preferences > General > Language
*** Apparently can only pick through languages here: /Applications/Opera.app/Contents/Resources/
** IE8: Tools > Internet Options > General > Appearance > Languages > Language Reference > Add...
*** You can also create a User-Defined language by adding the language code directly
** IE9: Tools > Internet Options > General > Appearance > Languages > Language Reference > Add...
*** You can also create a User-Defined language by adding the language code directly
** iOS/Safari: TBD
*** Note: This will change the locale (at least temporarily) for all apps on iPhone and/or iPad
* Quick verification of localized headers
** Here is a good site: http://www.ericgiguere.com/tools/http-header-viewer.html
*** Scroll down to the "Your Headers" section and look for "accept-language" row.
*** The ordered language list should show up there.
*** Example: eo,en-us;q=0.5
* OS-specific changes, if any:
** Ex: Rebooting to clear cache, data, cookies, etc.
** Ex: text/font encodings
** Custom language entries for browsers
=== Accessibility - Desktop only ===
* Verify minimal accessibility in the UI - keyboard only
** Creating a new BrowserID account
*** Email verification from mail application and from BrowserID
** Adding a new email to a current BrowserID account
*** Email verification from mail application and from BrowserID
** Changing/resetting a password - RP flow
*** Email verification from mail application and from BrowserID
** Changing a password - Account Manager
** Deleting an email from an account - Account Manager
** Deleting an account - Account Manager
** Other UI flows: This is not me, BrowserID links, etc.
* Note: this testing may require changes at the OS-level (like Mac OS) or changes at the browser level to more fully support keyboard-only access to a site.
=== Bug Regression ===
* Manual and automated testing of Verified/Closed bugs and issues from previous cycles of testing.
* The list of bugs to regress are always listed from the ChangeLog link in the deployment ticket.
* REF
** Server: https://github.com/mozilla/browserid/issues
** RPs/Clients: https://github.com/lloyd/myfavoritebeer.org/issues
** and https://github.com/mozilla/123done/issues
* Ideally, this regression would be in the form of a quick automated test/script to check key fixes/changes/updates.
* QA to work with Dev and PM to come up with a list of bugs/issues candidates to from the test cases for manual and automated testing.


== Bug Regression ==
* Tagging Regressions
Manual and automated testing of Verified/Closed bugs and issues from previous cycles of testing.
** If an old issue is found to exist in the code, then file a new issue and tag it with [regression] in the Title
Ideally, this would be in the form of a quick automated test/script to check key fixes/changes/updates.
** Give that regression issue a 5-star label
QA to work with Dev and PM to come up with a list of bugs/issues candidates to from the test cases for manual and automated testing.
* https://github.com/mozilla/browserid/issues
* https://github.com/mozilla/browserid/issues/milestones


== OS/Browser integration ==
=== OS/Browser integration ===
Manual testing/verification of the required OS and browser configurations:
Manual testing/verification of the required OS and browser configurations:
* Verify access and use across browsers on same OS, different OS
* Verify access and use across browsers on same OS, different OS
Line 199: Line 603:
* Browser synchronization - same platform, across platforms
* Browser synchronization - same platform, across platforms


== Mobile Testing ==
* Verify RP and main site access
Milestone is completed, so some Mobile specific tests should be added: TBD
** With and without third-party cookies enabled
* Sanity/Acceptance
** With and without all cookies enabled
* Functional/UI
 
* OS/Browser
* Verify cached headers, status, content across browsers
 
* Verify browser cookie information/inspection (per browser)
** Inspect the client and server browser cookie information for BrowserID specific information
** Privacy settings in browsers: start with 3rd-party cookies due to issues with eyedee.me
* Verify door hanger information (per browser, desktop and mobile)
** Inspect/verify the doorhanger information and links (if any)
 
* Also, QA needs to verify the clearing of all BrowserID data from a specific OS/browser/profile.
** See the following for more information:
** https://wiki.mozilla.org/QA/BrowserID/BrowserID_Weekly_Trains_Beta#Browser_and_Mobile_Support
 
* On the FireFox browser, various tools can be used to help test/verify BrowserID:
** Tools > Web Developer > Web Console: can be used to track live GET and POST messages from the RP (client) site.
** Tools > Web Developer > Error Console: to track any errors, warnings, or messages from the RP (client) site.
** From the mobile version: TBD
 
* On the Safari browser:
** Preferences > Advanced > "Show Develop menu in menu bar"
** Now, reference all tools through the Develop drop-down menu"
*** Error Console
** From the mobile version: Settings > Safari > Advanced > Debug Console > On
 
* On the Google Chrome browser:
** Tool icon > Tools > Developer Tools
** View > Developer > Developer Tools
 
* On the Opera browser:
** View > Developer Tools > Error Console
 
* On the IE8/IE9 browsers:
** F12 or Developer Tools
 
* On the Android Stock browser: TBD
 
=== BrowserID Logs/Messages ===
* Verify that the main BrowserID logs contain meaningful information for inactive and active messaging:
** browserid.log
*** QA is most concerned about any warnings or errors that come up during testing, and any debug messages that do not belong (talk to Dev about list of current debug messages)
** verifier.log
*** QA should not see any debug, warn, or error messages in this log (verify with Dev)
** keysigner.log
*** QA should not see any debug, warn, or error messages in this log (verify with Dev)
** dbwriter.log
*** QA should not see any debug, warn, or error messages in this log (verify with Dev)
** All *-error.log files should be empty. If they are not, this could point to an issue requiring debug and Dev/OPs interaction.
** NOTE: archived logs (in gzip format) may need to be reviewed if testing is generating a lot of log activity or if issues need to be debugged over two or more log rotations.
 
=== Mobile Testing ===
Mobile-specific tests should be added to verify compatibility/stability/functionality of BrowserID across Android and iOS tablets and phones
* Complete the Sanity/Acceptance section for phone, for tablet
* Complete the Basic Functional and UI sections on phone, on tablet
** Panning, zooming, scrolling, tab access and flow between tabs
* Complete the Specific RP/Client tests on phone, tablet
* OS/Browser interaction using FireFox and stock browser on Android phones and tablets
* OS/Browser interaction using Safari browser on iPhones and iPads
* Compatibility - review actions and processes that are unique to mobile devices
* Verify compatibility and functionality when working with Android/iOS Settings and with browser app-specific settings.
* Most of the testing will be based on the final matrix located here:
** https://docs.google.com/spreadsheet/ccc?key=0ArxosXOjCBacdHNrWFdBQmE3RU0xRFFwMUcxQWdRcXc&hl=en_US
 
=== Network/Internet ===
Verify the impact of various ways to access the internet: timing, communication, messaging, outages, errors, traffic, etc.
* Ethernet - personal vs. office, with and without VPN
* Public WiFi
* Private WiFi and other home setups
* 3g, 4g
 
=== Client-side LocalStorage ===
* Most, if not all, of the BrowserID-specific information is now kept on the server.
* But, LocalStorage should still be examined for any unexpected changes based on BrowserID use.
* Work with Dev to come up with meaningful tests to verify BrowserID's use of, and impact on, LocalStorage.
* Example issue: 688: IE8 blowing up trying to do "delete" on localStorage members.
* All session data is stored in the cookie (per Ben Adida).


== Client-side Testing ==
=== Server-side Testing ===
QA will need the local storage information per browser in order to verify accurate creation/update/use/removal of BrowserID data.
* Verify some UI elements and actions only found on the server
* Possible FireFox example:
** See UI section above.
** Users > USER-NAME > Library > Application Support > Firefox > Profiles > PROFILE.default
* Where is the local storage for BrowserID?
** Is it just localstore.rdf, or a whole set of files per profile?
* Local storage verification
** Per OS, per browser, per user/account
** Is this also further defined by profile?
*** OS > browser > profile > account/user


== Server-side Testing ==
Need the list of server and services or components that describe the BrowserID architecture to accurately focus on server-side testing.
* Verify some UI elements and actions only found on the server:
** Manage Accounts page: https://www.browserid.org/manage
** Users: https://browserid.org/users, as in https://browserid.org/users/<insert_mail_adress_registered>.xml
** https://browserid.org/verify
* Log inspection to verify accurate information
* Log inspection to verify accurate information
** Verify content of web server access logs
** See the Logging section above for more details
** Verify content of web server error logs
 
** Verify content of BrowserID logs
* Load Testing and Monitoring
* What other server-specific tests are required?
** See OPs section below for more information
* See also the Areas Not Covered section below for security and end-to-end (flows).


* API automation (if applicable/useful)


= Areas Not Covered or In Development =  
* Unit Tests (work with Dev)
== Server and Client Automation ==
 
Currently, QA has no automation configured for the weekly BrowserID testing.
* Headless frontend testing (work with Dev)
* Client-side automation using Selenium is being investigated
 
* TBD: Server-side automation, primarily to cover the API
* QA should look into headless (no browser/client) testing of server
 
* Server/service scenarios - failover/recover, temp outage, traffic flow, etc.
** Zeus/load balancer issues
** webhead for browserid, verifier, router, and static
** swebhead for dbwriter
** keysigner for keysigner
** mysql/db
 
=== Load Testing ===
* Load Testing in Stage is handled by QA using BrowserID code provided by Dev.
* QA should monitor all load tests
* QA should also schedule specific load tests to cover various requirements for server-side testing.
** Work with OPs to queue up specific load tests to check Server stability, various metrics
* Monitor load tests with OPs dashboards - e.g., http://pencil.scl2.stage.svc.mozilla.com/dash/stage/bid_webhead (requires access to the MPT VPN)
* Analyze text and visual data for functionality and peformance


== Security: VEP, VES, VEC ==
QA could use some help from Dev and PM on testable use cases.
* Valid/invalid public key
* Tracking public and private key usage
* Security/Crypto checks
* Damaged/Stolen/Compromised accounts, emails, keys
* Testing validity period for specific certificates
* Snooping email providers
* REF:
* REF:
** https://wiki.mozilla.org/Identity/Verified_Email_Protocol (is this the more current link?)
** load_gen: https://github.com/mozilla/browserid/blob/dev/bin/load_gen
** https://wiki.mozilla.org/Labs/Identity/VerifiedEmailProtocol
<pre>
** https://wiki.mozilla.org/Security/Reviews/Identity/browserid
An example command line to run the load test:
 
    nohup bin/load_gen -s https://verifier.login.anosrep.org -o -m 250000 -u 1/250
 
-s  -- the URL to the browserid server
-o  -- don't request static resources
-m  -- maximum active users to simulate (ADU)
-u  -- use this range of #@loadtest.domain users (these users already exist in the
      stage database. If running your own instance, do `CREATE_TEST_USERS=1000
      BROWSERID_FAKE_VERIFICATION=1 NODE_ENV=test_mysql npm start`).
 
I typically run the above command piped into this: 


== Flows/End-To-End ==
    | /usr/local/bin/tai64n | /usr/local/bin/tai64nlocal \
Verifying each step in the various flows and also the complete End-To-End flow. This testing will need to be done manually, if possible, for a few weekly trains to get the required tests and tools scoped out. Then, this can be moved over to automated E2E testing (alongside the automated smoke testing). This seems like a better fit for server-side automation, where the appropriate actions and verifications could be tested per flow.
      >> /path/to/load_gen.log-`date +%Y%m%d-%H%M` 2>&1 &
Important flows to test/validate:
* Certificate Provisioning: 8 steps to verify full certificate provisioning path
* Assertion Generation: 4 steps to verify assertion generation path
* Assertion Verification: 6 steps to verify assertion verification path
QA could use some help from PM and Dev here to better define use cases for each step of these flows. Provide ideas/cases for testing each of the steps, plus the included assumptions and requirements.
If applicable we should design both client-side and server-side tests.
QA could also leverage off the design/purpose/coverage of the unit tests.
REF: http://lloyd.io/how-browserid-works


== FireFox Add-On ==
which will timestamp the output and save it in a logfile, etc.
For Firefox users with the addon, the addon flow needs tested.
</pre>
Test design: TBD


== Internationalization, Localization ==
== QA Sign-Off for Stage ==
* Localization (L10n) requirements TBD
* Requirements and procedures:
* Internationalization (i10n) requirements TBD
** Complete all required testing for the current weekly train: resolved/closed issues, suggested areas for QA focus, specific features and areas of test coverage, automation, etc.
** Update the deployment ticket with a summary of the completed testing, plus the following: issues verified, issues opened, issues reopened, any testing marked incomplete, delayed, or removed from this release, etc.
** Add a sign-off to the deployment ticket if this weekly release passes QA
** Add additional comments (without a sign-off) if this weekly release fails QA


= Setting Up Test Emails =
== QA Testing for Production Pushes ==
== Email-side (PIA) ==
This is a work in progress!...
* This is the Primary Identity Authority (PIA) for all the transactions.
* Examples for testing: Mozilla, Yahoo, Google, ISPs
* Also available for testing are sites like Mailinator
== Using Internal Email Accounts ==
* If secure enough to do so: <mozilla name>+tag@mozilla.com
* Otherwise: Create emails at www.mailinator.com
* Do we have other tested (re: reliable) methods for creating test/dummy email accounts?


* Verify push to PHX1 (/etc/hosts hack)
** Standard sanity testing across platforms, OS, and browsers
* Verify push to SCL2 (/etc/hosts hack)
** Standard sanity testing across platforms, OS, and browsers
* Locale verification
** Based on final pull request for config/l10n-prod.json
** Quick sanity test of new/changed/removed locales
* Check the verifier service before OPs undrains a datacenter during a push (for PHX1 and SCL2)
** REF: Bug 753728 - browserid.org/verify is on occasion throwing 500 errors (being caught on the load balancer)
** REF: Bug 753828 - monitor the state of zeus pools


= Open Questions/Issues for QA, Dev, and PM =
== QA Sign-Off for Production ==
# How do we verify this statement: "Unlike  other sign-in systems, BrowserID does not leak information back to any  server (not even to the BrowserID servers) about which sites a user  visits." Could use some help from Dev and PM on testable use cases.
* TBD: Requirements and procedures:
# How does this site/technology fit in? Or is it not applicable to our weekly testing?
* Similar to sign-off for Stage, but applies to both data centers
http://people.mozilla.com/~faaborg/files/projects/firefoxAccount/index.html
** Add the Services Maintenance Window information to the deployment ticket.
# Is there any benefit to testing/comparing BrowserID with OpenID or any other ID mechanism?
** Complete all required testing for the push to Production.
# Where is local storage for BrowserID? What data is stored per browser/profile/account/user?
** Update the deployment ticket with a summary of the completed testing.
# What is the best way to approach security and privacy testing? Through the UI? through the API? Or, by setting up some unsecure environment or hackable instance?
** Add a sign-off to the deployment ticket if this Production push passes QA.
# Is it possible to have multiple accounts referencing the same email(s)?
** Add additional comments (without a sign-off) if this Production push fails QA.
# What about multiple accounts per user (for work/prof/public emails vs. private/personal emails)?
# Do we need to test/verify deviations from the standard VEC?




Back To ==> [[QA/Browser_Technologies/Services | Mozilla BT Services]]
Back To ==> [[QA/Browser_Technologies/Services | Mozilla BT Services]]
Edwong
Confirmed users
964

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/344137...672405"