Confirmed users
964
edits
QA/BrowserID/TestPlan: Difference between revisions
→Sanity/Acceptance/Smoke
| (13 intermediate revisions by 4 users not shown) | |||
| Line 41: | Line 41: | ||
** https://docs.google.com/spreadsheet/ccc?key=0ArxosXOjCBacdHNrWFdBQmE3RU0xRFFwMUcxQWdRcXc&hl=en_US | ** https://docs.google.com/spreadsheet/ccc?key=0ArxosXOjCBacdHNrWFdBQmE3RU0xRFFwMUcxQWdRcXc&hl=en_US | ||
* Test Cases in Moztrap: https://moztrap.allizom.org/manage/cases/?filter-productversion=2 | * Test Cases in Moztrap: | ||
and here: https://moztrap.mozilla.org/manage/cases/?filter-suite=82 | ** https://moztrap.allizom.org/manage/cases/?filter-productversion=2 | ||
** and here: https://moztrap.mozilla.org/manage/cases/?filter-suite=82 | |||
=== Weekly Test Schedules === | === Weekly Test Schedules === | ||
| Line 53: | Line 54: | ||
=== Weekly Meetings === | === Weekly Meetings === | ||
* BrowserID Team Meeting: Mondays, | * BrowserID Team Meeting: Mondays, 10:30am PST/PDT in Vidyo Rm: Identity | ||
** Wiki: https://wiki.mozilla.org/Identity/WeeklyMeeting | ** Wiki: https://wiki.mozilla.org/Identity/WeeklyMeeting | ||
* BrowserID Bug Triage Meeting: Thursdays, 9:30am PST/PDT in | * BrowserID Bug Triage Meeting: Thursdays, 9:30am PST/PDT in Vidyo Rm: Identity | ||
** https://github.com/mozilla/browserid/issues | ** https://github.com/mozilla/browserid/issues | ||
| Line 225: | Line 226: | ||
** https://wiki.mozilla.org/QA/BrowserID/BrowserID_Basic_Install#Running_Unit_Tests | ** https://wiki.mozilla.org/QA/BrowserID/BrowserID_Basic_Install#Running_Unit_Tests | ||
* Instructions are good for any installation... | * Instructions are good for any installation... | ||
=== Automation Testing === | |||
* There are two main ways to run automation testing: | |||
** Directly through the Selenium/Jenkins/Sauce Labs automation environment | |||
** Locally using a stand-alone Selenium server and the built-in automation tests | |||
* Selenium/Jenkins/Sauce Labs automation - see the following links | |||
** https://wiki.mozilla.org/QA/BrowserID/BrowserID_Basic_Install#Running_the_Automation_Tests | |||
** https://wiki.mozilla.org/Identity/QA#BrowserID_Automation | |||
*** Selenium BID dashboard: BrowserID dashboard: http://qa-selenium.mv.mozilla.com:8080/ | |||
*** Browserid-tests repo: https://github.com/mozilla/browserid/tree/dev/automation-tests | |||
*** Dave Hunt's BIDPOM (BrowserID Page-Object Model) repo: https://github.com/davehunt/bidpom | |||
* Automation on the localhost - see the following links: | |||
** https://wiki.mozilla.org/QA/BrowserID/BrowserID_Basic_Install#Running_the_Automation_Tests | |||
=== Sanity/Acceptance/Smoke === | === Sanity/Acceptance/Smoke === | ||
| Line 231: | Line 247: | ||
* See the Test Cases (listed in the "Links and Documentation" section above) for more information. | * See the Test Cases (listed in the "Links and Documentation" section above) for more information. | ||
* Server-side (https://login.anosrep.org/) [https://moztrap. | * Server-side (https://login.anosrep.org/) [https://moztrap.mozilla.org/manage/cases/?filter-productversion=2&filter-suite=82 MozTrap] [https://github.com/mozilla/browserid/blob/dev/automation-tests/persona_server/tests/test_manage_account.py AUTO] | ||
** [https://moztrap.mozilla.org/manage/case/989/ Sign Up with a new account and one email] | |||
** [https://moztrap. | ** [https://moztrap.mozilla.org/manage/case/990/ Sign In - Sign Out with a current account of one email] | ||
** [https://moztrap. | ** [https://moztrap.mozilla.org/manage/case/992/ Change the password on a current account with one email] | ||
** [https://moztrap. | ** [https://moztrap.mozilla.org/manage/case/993/ Delete/Remove a current account with one email] | ||
** [https://moztrap. | |||
* Client-side (http://myfavoritebeer.org or http://123done.org using one desktop OS with one browser) | * Client-side (http://myfavoritebeer.org or http://123done.org using one desktop OS with one browser) | ||
| Line 250: | Line 265: | ||
** Delete/Remove the account from the server-side | ** Delete/Remove the account from the server-side | ||
* Client-side (http://myfavoritebeer.org or http://123done.org using Android with stock browser) | * Client-side (http://beta.myfavoritebeer.org or http://beta.123done.org using Android with stock browser) | ||
** Sign In using one RP/client to create a new account with one email | ** Sign In using one RP/client to create a new account with one email | ||
*** Verify correct email verification sent to correct account (email provider) | *** Verify correct email verification sent to correct account (email provider) | ||
| Line 272: | Line 287: | ||
* Site Access - Stage | * Site Access - Stage | ||
** Verify use of HSTS (for https:// only sites) | ** Verify use of HSTS (for https:// only sites) | ||
** Verify that the Stage server site can be accessed through secured HTTPS: https:// | ** Verify that the Stage server site can be accessed through secured HTTPS: https://login.anosrep.org/ | ||
** Verify that the Stage server site can not be accessed using only HTTP, but instead gets redirected to the HTTPS site: http:// | ** Verify that the Stage server site can not be accessed using only HTTP, but instead gets redirected to the HTTPS site: http://login.anosrep.org/ | ||
** Verify that the Stage RP/client site can be accessed through standard HTTP: http://beta.myfavoritebeer.org/ or http://123done.org | ** Verify that the Stage RP/client site can be accessed through standard HTTP: http://beta.myfavoritebeer.org/ or http://beta.123done.org | ||
** Verify that the Stage RP/client site gets a certificate warning/error or is blocked when accessed through secured HTTPS: https://beta.myfavoritebeer.org/, http://123done.org | ** Verify that the Stage RP/client site gets a certificate warning/error or is blocked when accessed through secured HTTPS: https://beta.myfavoritebeer.org/, http://beta.123done.org | ||
** Verify similar for the following RPs/clients: | ** Verify similar for the following RPs/clients: | ||
*** carrera.databits.net:9999/ | *** carrera.databits.net:9999/ | ||
| Line 692: | Line 707: | ||
** mysql/db | ** mysql/db | ||
=== | === Load Testing === | ||
* Load Testing in Stage is handled by | * Load Testing in Stage is handled by QA using BrowserID code provided by Dev. | ||
* QA should monitor all load tests | * QA should monitor all load tests | ||
* QA should also schedule specific load tests to cover various requirements for server-side testing. | * QA should also schedule specific load tests to cover various requirements for server-side testing. | ||
** Work with OPs to queue up specific load tests to check Server stability, various metrics | ** Work with OPs to queue up specific load tests to check Server stability, various metrics | ||
* Monitor load tests with OPs dashboards | * Monitor load tests with OPs dashboards - e.g., http://pencil.scl2.stage.svc.mozilla.com/dash/stage/bid_webhead (requires access to the MPT VPN) | ||
* Analyze text and visual data for functionality and peformance | * Analyze text and visual data for functionality and peformance | ||
* REF: | * REF: | ||
** load_gen: https://github.com/mozilla/browserid/blob/dev/bin/load_gen | ** load_gen: https://github.com/mozilla/browserid/blob/dev/bin/load_gen | ||
<pre> | |||
An example command line to run the load test: | |||
nohup bin/load_gen -s https://verifier.login.anosrep.org -o -m 250000 -u 1/250 | |||
-s -- the URL to the browserid server | |||
-o -- don't request static resources | |||
-m -- maximum active users to simulate (ADU) | |||
-u -- use this range of #@loadtest.domain users (these users already exist in the | |||
stage database. If running your own instance, do `CREATE_TEST_USERS=1000 | |||
BROWSERID_FAKE_VERIFICATION=1 NODE_ENV=test_mysql npm start`). | |||
I typically run the above command piped into this: | |||
| /usr/local/bin/tai64n | /usr/local/bin/tai64nlocal \ | |||
>> /path/to/load_gen.log-`date +%Y%m%d-%H%M` 2>&1 & | |||
which will timestamp the output and save it in a logfile, etc. | |||
</pre> | |||
== QA Sign-Off for Stage == | == QA Sign-Off for Stage == | ||