MozillaWiki

QA/BrowserID/TestPlan: Difference between revisions

Page Discussion

QA/BrowserID/TestPlan (view source)

Revision as of 01:29, 3 July 2013

2,020 bytes added ,  3 July 2013
→‎Sanity/Acceptance/Smoke
 
(30 intermediate revisions by 4 users not shown)
Line 41: Line 41:
** https://docs.google.com/spreadsheet/ccc?key=0ArxosXOjCBacdHNrWFdBQmE3RU0xRFFwMUcxQWdRcXc&hl=en_US
** https://docs.google.com/spreadsheet/ccc?key=0ArxosXOjCBacdHNrWFdBQmE3RU0xRFFwMUcxQWdRcXc&hl=en_US


* BrowserID Test Cases: Set 1
* Test Cases in Moztrap:  
** https://docs.google.com/document/d/1fmT-Egj4-pJeQBKJiHw_eIIOkR-TmW7SE1yeXgbqF3U/edit?pli=1
** https://moztrap.allizom.org/manage/cases/?filter-productversion=2
** and here: https://moztrap.mozilla.org/manage/cases/?filter-suite=82


=== Weekly Test Schedules ===
=== Weekly Test Schedules ===
Line 53: Line 54:


=== Weekly Meetings ===
=== Weekly Meetings ===
* BrowserID Team Meeting: Mondays, 9:30am PST/PDT in 2-CYA
* BrowserID Team Meeting: Mondays, 10:30am PST/PDT in Vidyo Rm: Identity
** Wiki: https://wiki.mozilla.org/Identity/WeeklyMeeting
** Wiki: https://wiki.mozilla.org/Identity/WeeklyMeeting
* BrowserID Bug Triage Meeting: Thursdays, 9:30am PST/PDT in 2-JK
* BrowserID Bug Triage Meeting: Thursdays, 9:30am PST/PDT in Vidyo Rm: Identity
** https://github.com/mozilla/browserid/issues
** https://github.com/mozilla/browserid/issues


Line 70: Line 71:
** benadida = Ben Adida developer
** benadida = Ben Adida developer
** thunder = Dan Mills PM
** thunder = Dan Mills PM
** petef = Pete Fritchman OPs
** gene = Gene Wood OPs
** jbonacci = James Bonacci QA
** jbonacci = James Bonacci QA
** jrgm = John Morrison QA
** jrgm = John Morrison QA
Line 76: Line 77:


== Bugs and Open Issues ==
== Bugs and Open Issues ==
* Infrastructure Bugs in Bugzilla
* Security, OPs, and Apps-related bugs in Bugzilla
** Classification: Other
** Classification: Client Software
** Product: mozilla.org
** Component: Server: Identity
** Component: Server Operations: Labs
** Product: Mozilla Services
*** Note: This includes bugs for any changes to the Stage environment and the weekly release deployment ticket.
* Deployment Requests in Bugzilla
** Classification: Client Software
** Product: Mozilla Services
** Component: Operations: Deployment Requests


* BrowserID RP/Client and Server Bugs/Issues
* BrowserID RP/Client and Server Bugs/Issues
** GitHub server: https://github.com/mozilla/browserid/issues
** GitHub server: https://github.com/mozilla/browserid/issues
** GitHub RP/client: https://github.com/mozilla/123done/issues
** GitHub RP/client: https://github.com/lloyd/myfavoritebeer.org/issues
** GitHub RP/client: https://github.com/lloyd/myfavoritebeer.org/issues


Line 101: Line 106:


== RP/Client and Server Test Environments ==  
== RP/Client and Server Test Environments ==  
* Stage Environment (OPs build-out):
* Stage Environment:
** Server: https://diresworb.org/
** Server: https://login.anosrep.org/ (https://anosrep.org, was https://diresworb.org/)
** RP/Client 1: http://beta.myfavoritebeer.org/
** RP/Client 1: http://beta.myfavoritebeer.org/
** RP/Client 2: http://beta.myfavoritebooze.org/ (OBSOLETE)
** RP/Client 2: http://beta.123done.org/
** RP/Client 3: http://beta.myfavoriteshow.org/ (OBSOLETE)
** RP/Client 4: http://beta.123done.org/
** RP/Client 5: http://carrera.databits.net:9999/ (supports the OPs' sinatra-browserid plugin)
** Primary 1: https://eyedee.me/
** Primary 1: https://eyedee.me/


* Dev Environment (for comparison/verification against Stage or Prod)
* Dev Environment:
** Server: https://dev.diresworb.org/
** Server: https://login.dev.anosrep.org (https://dev.anosrep.org, was https://dev.diresworb.org/)
** RP/Client 1: http://dev.myfavoritebeer.org/
** RP/Client 1: http://dev.myfavoritebeer.org/
** RP/Client 2: http://dev.myfavoritebooze.org/ (OBSOLETE)
** RP/Client 2: http://dev.123done.org/
** RP/Client 3: http://dev.myfavoriteshow.org/ (OBSOLETE)
** RP/Client 4: http://dev.123done.org/
** Primary 1: https://eyedee.me/
** Primary 1: https://eyedee.me/


* Production Environment (for comparison/verification against Dev or Stage)
* Production Environment:
** Server: https://browserid.org/
** Server: https://login.persona.org (https://persona.org, was https://browserid.org/)
** RP/Client 1: http://www.myfavoritebeer.org/
** RP/Client 1: http://www.myfavoritebeer.org/
** RP/Client 2: http://www.myfavoritebooze.org/ (OBSOLETE)
** RP/Client 2: http://123done.org/
** RP/Client 3: http://www.myfavoriteshow.org/ (OBSOLETE)
** Other working RPs/Clients: any Mozilla internal or public site using Persona/BrowserID
** RP/Client 4: http://123done.org/
** Other working RPs/Clients: any Mozilla internal or public site using BID
** Primary 1: https://eyedee.me/
** Primary 1: https://eyedee.me/
** Primary 2: https://mockmyid.com/
** Primary 2: https://mockmyid.com/
Line 140: Line 138:
* webheads: /var/log/browserid
* webheads: /var/log/browserid
** browserid.log
** browserid.log
** browserid-metrics.json
** router.log
** router.log
** verifier.log
** verifier.log
** verifier-metrics.json
** verifier-metrics.json
** verifier-compute.log
** verifier-compute.log
** static.log
** router-metrics.json
Note: router-metrics.json replaces browserid-metrics.json
* secure webheads: /var/log/browserid/
* secure webheads: /var/log/browserid/
** dbwriter.log
** dbwriter.log
* keysigners have /var/log/browserid/
* keysigners have /var/log/browserid/
** keysigner.log
** keysigner.log
** keysigner-compute.log
** keysigner-compute.log
* dbs: various db logs
* dbs: various db logs
* zeus: various zeus logs
* zeus: various zeus logs


Line 175: Line 178:
*** node bin/verifier
*** node bin/verifier
*** node bin/router
*** node bin/router
*** node bin/static
*** /usr/bin/node /opt/browserid/lib/verifier/verifier-compute.js
*** /usr/bin/node /opt/browserid/lib/verifier/verifier-compute.js
*** /usr/bin/node /opt/browserid/lib/bcrypt-compute.js
*** /usr/bin/node /opt/browserid/lib/bcrypt-compute.js
** secure webheads:
** secure webheads:
*** node bin/dbwriter
*** node bin/dbwriter
*** /usr/bin/node /opt/browserid/lib/bcrypt-compute.js
*** /usr/bin/node /opt/browserid/lib/bcrypt-compute.js
** keysigners:
** keysigners:
*** node bin/keysigner
*** node bin/keysigner
*** /usr/bin/node /opt/browserid/lib/keysigner/keysigner-compute.js
*** /usr/bin/node /opt/browserid/lib/keysigner/keysigner-compute.js
** databases:
** databases:
*** /bin/sh /usr/bin/mysqld_safe --datadir=/data/mysql --pid-file=/var/run/mysqld/mysqld.pid
*** /bin/sh /usr/bin/mysqld_safe --datadir=/data/mysql --pid-file=/var/run/mysqld/mysqld.pid
*** /usr/sbin/mysqld --basedir=/usr --datadir=/data/mysql --user=mysql --log-error=/data/mysql/db1.iddb.scl2.stage.svc.mozilla.com.err --open-files-limit=8192 --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
*** /usr/sbin/mysqld --basedir=/usr --datadir=/data/mysql --user=mysql --log-error=/data/mysql/db1.iddb.scl2.stage.svc.mozilla.com.err --open-files-limit=8192 --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
*** perl /usr/bin/pt-heartbeat --create-table -D browserid -u root --update
*** perl /usr/bin/pt-heartbeat --create-table -D browserid -u root --update
** Note: This assumes access to the main Stage Admin server. If you do not have SSH access, then get QA staff or Dev to verify the processes.
** Note: This assumes access to the main Stage Admin server. If you do not have SSH access, then get QA staff or Dev to verify the processes.


* Version Checks
* Version Checks
** Production: https://browserid.org/ver.txt
** Production: https://login.persona.org/ver.txt
** Dev: https://dev.diresworb.org/ver.txt
** Dev: https://login.dev.anosrep.org/ver.txt
** Stage: https://diresworb.org/ver.txt
** Stage: https://login.anosrep.org/ver.txt
** Note: Dev should always be running one version newer than Stage. Stage should be running one version newer than Production.
** Note: Dev should always be running one version newer than Stage. Stage should be running one version newer than Production.


Line 198: Line 206:
** See Stage Logging section above.
** See Stage Logging section above.
** Note: This assumes access to the main Stage Admin server. If you do not have SSH access, then get QA staff or Dev to verify that the logs are active.
** Note: This assumes access to the main Stage Admin server. If you do not have SSH access, then get QA staff or Dev to verify that the logs are active.
* Verify heartbeat on the servers
** Dev: https://dev.diresworb.org/__heartbeat__
** Prod: https://browserid.org/__heartbeat__
** Stage: https://diresworb.org/__heartbeat__
** all of them should return "ok" to the browser


=== Local Installs ===
=== Local Installs ===
Line 224: Line 226:
** https://wiki.mozilla.org/QA/BrowserID/BrowserID_Basic_Install#Running_Unit_Tests
** https://wiki.mozilla.org/QA/BrowserID/BrowserID_Basic_Install#Running_Unit_Tests
* Instructions are good for any installation...
* Instructions are good for any installation...
=== Automation Testing ===
* There are two main ways to run automation testing:
** Directly through the Selenium/Jenkins/Sauce Labs automation environment
** Locally using a stand-alone Selenium server and the built-in automation tests
* Selenium/Jenkins/Sauce Labs automation - see the following links
** https://wiki.mozilla.org/QA/BrowserID/BrowserID_Basic_Install#Running_the_Automation_Tests
** https://wiki.mozilla.org/Identity/QA#BrowserID_Automation
*** Selenium BID dashboard: BrowserID dashboard: http://qa-selenium.mv.mozilla.com:8080/
*** Browserid-tests repo: https://github.com/mozilla/browserid/tree/dev/automation-tests
*** Dave Hunt's BIDPOM (BrowserID Page-Object Model) repo: https://github.com/davehunt/bidpom
* Automation on the localhost - see the following links:
** https://wiki.mozilla.org/QA/BrowserID/BrowserID_Basic_Install#Running_the_Automation_Tests


=== Sanity/Acceptance/Smoke ===
=== Sanity/Acceptance/Smoke ===
Line 230: Line 247:
* See the Test Cases (listed in the "Links and Documentation" section above) for more information.
* See the Test Cases (listed in the "Links and Documentation" section above) for more information.


* Server-side (https://diresworb.org/)
* Server-side (https://login.anosrep.org/) [https://moztrap.mozilla.org/manage/cases/?filter-productversion=2&filter-suite=82 MozTrap] [https://github.com/mozilla/browserid/blob/dev/automation-tests/persona_server/tests/test_manage_account.py AUTO]
** Sign Up with a new account and one email
** [https://moztrap.mozilla.org/manage/case/989/ Sign Up with a new account and one email]
*** Verify correct email verification sent to correct account (email provider)
** [https://moztrap.mozilla.org/manage/case/990/ Sign In - Sign Out with a current account of one email]
** Sign In with a current account of one email
** [https://moztrap.mozilla.org/manage/case/992/ Change the password on a current account with one email]
** Sign Out from a current account of one email
** [https://moztrap.mozilla.org/manage/case/993/ Delete/Remove a current account with one email]
** Change the password on a current account with one email
*** Verify correct email verification sent to correct account (email provider)
** Delete/Remove a current account with one email


* Client-side (http://myfavoritebeer.org or http://123done.org using one desktop OS with one browser)
* Client-side (http://myfavoritebeer.org or http://123done.org using one desktop OS with one browser)
Line 251: Line 265:
** Delete/Remove the account from the server-side
** Delete/Remove the account from the server-side


* Client-side (http://myfavoritebeer.org or http://123done.org using Android with stock browser)
* Client-side (http://beta.myfavoritebeer.org or http://beta.123done.org using Android with stock browser)
** Sign In using one RP/client to create a new account with one email
** Sign In using one RP/client to create a new account with one email
*** Verify correct email verification sent to correct account (email provider)
*** Verify correct email verification sent to correct account (email provider)
Line 273: Line 287:
* Site Access - Stage
* Site Access - Stage
** Verify use of HSTS (for https:// only sites)
** Verify use of HSTS (for https:// only sites)
** Verify that the Stage server site can be accessed through secured HTTPS: https://www.diresworb.org/
** Verify that the Stage server site can be accessed through secured HTTPS: https://login.anosrep.org/
** Verify that the Stage server site can not be accessed using only HTTP, but instead gets redirected to the HTTPS site: http://www.diresworb.org/
** Verify that the Stage server site can not be accessed using only HTTP, but instead gets redirected to the HTTPS site: http://login.anosrep.org/
** Verify that the Stage RP/client site can be accessed through standard HTTP: http://beta.myfavoritebeer.org/ or http://123done.org
** Verify that the Stage RP/client site can be accessed through standard HTTP: http://beta.myfavoritebeer.org/ or http://beta.123done.org
** Verify that the Stage RP/client site gets a certificate warning/error or is blocked when accessed through secured HTTPS: https://beta.myfavoritebeer.org/, http://123done.org
** Verify that the Stage RP/client site gets a certificate warning/error or is blocked when accessed through secured HTTPS: https://beta.myfavoritebeer.org/, http://beta.123done.org
** Verify similar for the following RPs/clients:
** Verify similar for the following RPs/clients:
*** carrera.databits.net:9999/
*** carrera.databits.net:9999/
Line 282: Line 296:
* Accounts and Email Verification
* Accounts and Email Verification
** Creating an account from the server site using Sign Up
** Creating an account from the server site using Sign Up
** Creating an account inline (at first use of an RP/client Sign In)
** Creating an account inline (at first use of an RP/client Sign In) [https://github.com/mozilla/BrowserID-Tests/blob/master/123done/tests/test_new_user.py AUTO]
** Email notifications for new accounts: verification email through email provider with proper email account listed, live verification link, etc.
** Email notifications for new accounts: verification email through email provider with proper email account listed, live verification link, etc. [https://github.com/mozilla/BrowserID-Tests/blob/master/123done/tests/test_new_user.py AUTO]
*** Test with emails/accounts on various, popular email servers/services
*** Test with emails/accounts on various, popular email servers/services
*** See more details in the next bulleted list
*** See more details in the next bulleted list
Line 291: Line 305:
* Email Notifications for server or RP/Client
* Email Notifications for server or RP/Client
** On the same OS
** On the same OS
*** Email notification and verification using one browser
*** Email notification and verification using one browser [https://github.com/mozilla/BrowserID-Tests/blob/master/123done/tests/test_new_user.py AUTO]
*** Email notification on one browser, verification on an another
*** Email notification on one browser, verification on an another
** Across multiple OS
** Across multiple OS
Line 312: Line 326:


* Accounts and Emails
* Accounts and Emails
** Adding additional emails to an account
** Adding additional emails to an account [https://github.com/mozilla/BrowserID-Tests/blob/master/123done/tests/test_add_another_email.py AUTO]
** Attempt to add an email (that may or may not be yours) from another account
** Attempt to add an email (that may or may not be yours) from another account
** Deleting one or more emails (without actually deleting the account)
** Deleting one or more emails (without actually deleting the account)
Line 688: Line 702:
* Server/service scenarios - failover/recover, temp outage, traffic flow, etc.
* Server/service scenarios - failover/recover, temp outage, traffic flow, etc.
** Zeus/load balancer issues
** Zeus/load balancer issues
** webhead for browserid and verifier
** webhead for browserid, verifier, router, and static
** swebhead for dbwriter
** swebhead for dbwriter
** keysigner for keysigner
** keysigner for keysigner
** mysql/db
** mysql/db


=== OPs Load Testing ===
=== Load Testing ===
* Load Testing in Stage is handled by OPs using BrowserID code provided by Dev.
* Load Testing in Stage is handled by QA using BrowserID code provided by Dev.
* QA should monitor all load tests run by OPs
* QA should monitor all load tests
* QA should also schedule specific load tests to cover various requirements for server-side testing.
* QA should also schedule specific load tests to cover various requirements for server-side testing.
** Work with OPs to queue up specific load tests to check Server stability, various metrics
** Work with OPs to queue up specific load tests to check Server stability, various metrics
* Monitor load tests with OPs dashboards
* Monitor load tests with OPs dashboards - e.g., http://pencil.scl2.stage.svc.mozilla.com/dash/stage/bid_webhead (requires access to the MPT VPN)
* Analyze text and visual data for functionality and peformance
* Analyze text and visual data for functionality and peformance


* REF:
* REF:
** load_gen: https://github.com/mozilla/browserid/blob/dev/bin/load_gen
** load_gen: https://github.com/mozilla/browserid/blob/dev/bin/load_gen
<pre>
An example command line to run the load test:
    nohup bin/load_gen -s https://verifier.login.anosrep.org -o -m 250000 -u 1/250
-s  -- the URL to the browserid server
-o  -- don't request static resources
-m  -- maximum active users to simulate (ADU)
-u  -- use this range of #@loadtest.domain users (these users already exist in the
      stage database. If running your own instance, do `CREATE_TEST_USERS=1000
      BROWSERID_FAKE_VERIFICATION=1 NODE_ENV=test_mysql npm start`).
I typically run the above command piped into this: 
    | /usr/local/bin/tai64n | /usr/local/bin/tai64nlocal \
      >> /path/to/load_gen.log-`date +%Y%m%d-%H%M` 2>&1 &
which will timestamp the output and save it in a logfile, etc.
</pre>


== QA Sign-Off for Stage ==
== QA Sign-Off for Stage ==
Edwong
Confirmed users
964

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/443750...672405"