MozillaWiki

QA/BrowserID/TestPlan: Difference between revisions

Page Discussion

QA/BrowserID/TestPlan (view source)

Revision as of 01:29, 3 July 2013

1,440 bytes added ,  3 July 2013
→‎Sanity/Acceptance/Smoke
 
(45 intermediate revisions by 4 users not shown)
Line 41: Line 41:
** https://docs.google.com/spreadsheet/ccc?key=0ArxosXOjCBacdHNrWFdBQmE3RU0xRFFwMUcxQWdRcXc&hl=en_US
** https://docs.google.com/spreadsheet/ccc?key=0ArxosXOjCBacdHNrWFdBQmE3RU0xRFFwMUcxQWdRcXc&hl=en_US


* BrowserID Test Cases: Set 1
* Test Cases in Moztrap:  
** https://docs.google.com/document/d/1fmT-Egj4-pJeQBKJiHw_eIIOkR-TmW7SE1yeXgbqF3U/edit?pli=1
** https://moztrap.allizom.org/manage/cases/?filter-productversion=2
** and here: https://moztrap.mozilla.org/manage/cases/?filter-suite=82


=== Weekly Test Schedules ===
=== Weekly Test Schedules ===
Line 53: Line 54:


=== Weekly Meetings ===
=== Weekly Meetings ===
* BrowserID Team Meeting: Mondays, 9:30am PST/PDT in 2-CYA
* BrowserID Team Meeting: Mondays, 10:30am PST/PDT in Vidyo Rm: Identity
** Wiki: https://wiki.mozilla.org/Identity/WeeklyMeeting
** Wiki: https://wiki.mozilla.org/Identity/WeeklyMeeting
* BrowserID Bug Triage Meeting: Thursdays, 9:30am PST/PDT in 2-JK
* BrowserID Bug Triage Meeting: Thursdays, 9:30am PST/PDT in Vidyo Rm: Identity
** https://github.com/mozilla/browserid/issues
** https://github.com/mozilla/browserid/issues


Line 70: Line 71:
** benadida = Ben Adida developer
** benadida = Ben Adida developer
** thunder = Dan Mills PM
** thunder = Dan Mills PM
** petef = Pete Fritchman OPs
** gene = Gene Wood OPs
** jbonacci = James Bonacci QA
** jbonacci = James Bonacci QA
** jrgm = John Morrison QA
** jrgm = John Morrison QA
Line 76: Line 77:


== Bugs and Open Issues ==
== Bugs and Open Issues ==
* Infrastructure Bugs in Bugzilla
* Security, OPs, and Apps-related bugs in Bugzilla
** Classification: Other
** Classification: Client Software
** Product: mozilla.org
** Component: Server: Identity
** Component: Server Operations: Labs
** Product: Mozilla Services
*** Note: This includes bugs for any changes to the Stage environment and the weekly release deployment ticket.
* Deployment Requests in Bugzilla
** Classification: Client Software
** Product: Mozilla Services
** Component: Operations: Deployment Requests


* BrowserID RP/Client and Server Bugs/Issues
* BrowserID RP/Client and Server Bugs/Issues
** GitHub server: https://github.com/mozilla/browserid/issues
** GitHub server: https://github.com/mozilla/browserid/issues
** GitHub RP/client: https://github.com/mozilla/123done/issues
** GitHub RP/client: https://github.com/lloyd/myfavoritebeer.org/issues
** GitHub RP/client: https://github.com/lloyd/myfavoritebeer.org/issues


Line 101: Line 106:


== RP/Client and Server Test Environments ==  
== RP/Client and Server Test Environments ==  
* Stage Environment (OPs build-out):
* Stage Environment:
** Server: https://diresworb.org/
** Server: https://login.anosrep.org/ (https://anosrep.org, was https://diresworb.org/)
** RP/Client 1: http://beta.myfavoritebeer.org/ (DEPRECATED)
** RP/Client 1: http://beta.myfavoritebeer.org/
** RP/Client 2: http://beta.myfavoritebooze.org/ (OBSOLETE)
** RP/Client 2: http://beta.123done.org/
** RP/Client 3: http://beta.myfavoriteshow.org/ (OBSOLETE)
** RP/Client 4: http://123done.org/
** RP/Client 5: http://carrera.databits.net:9999/ (supports the OPs' sinatra-browserid plugin)
** Primary 1: https://eyedee.me/
** Primary 1: https://eyedee.me/


* Dev Environment (for comparison/verification against Stage or Prod)
* Dev Environment:
** Server: https://dev.diresworb.org/
** Server: https://login.dev.anosrep.org (https://dev.anosrep.org, was https://dev.diresworb.org/)
** RP/Client 1: http://dev.myfavoritebeer.org/ (DEPRECATED)
** RP/Client 1: http://dev.myfavoritebeer.org/
** RP/Client 2: http://dev.myfavoritebooze.org/ (OBSOLETE)
** RP/Client 2: http://dev.123done.org/
** RP/Client 3: http://dev.myfavoriteshow.org/ (OBSOLETE)
** Primary 1: https://eyedee.me/
** Primary 1: https://eyedee.me/


* Production Environment (for comparison/verification against Dev or Stage)
* Production Environment:
** Server: https://browserid.org/
** Server: https://login.persona.org (https://persona.org, was https://browserid.org/)
** RP/Client 1: http://www.myfavoritebeer.org/ (DEPRECATED)
** RP/Client 1: http://www.myfavoritebeer.org/
** RP/Client 2: http://www.myfavoritebooze.org/ (OBSOLETE)
** RP/Client 2: http://123done.org/
** RP/Client 3: http://www.myfavoriteshow.org/ (OBSOLETE)
** Other working RPs/Clients: any Mozilla internal or public site using Persona/BrowserID
** Other working RPs/Clients: any Mozilla internal or public site using BID
** Primary 1: https://eyedee.me/
** Primary 1: https://eyedee.me/
** Primary 2: https://mockmyid.com/


== Supported OS and Browsers ==
== Supported OS and Browsers ==
Line 137: Line 138:
* webheads: /var/log/browserid
* webheads: /var/log/browserid
** browserid.log
** browserid.log
** browserid-metrics.json
** router.log
** verifier.log
** verifier.log
** verifier-metrics.json
** verifier-metrics.json
** verifier-compute.log
** verifier-compute.log
** static.log
** router-metrics.json
Note: router-metrics.json replaces browserid-metrics.json
* secure webheads: /var/log/browserid/
* secure webheads: /var/log/browserid/
** dbwriter.log
** dbwriter.log
* keysigners have /var/log/browserid/
* keysigners have /var/log/browserid/
** keysigner.log
** keysigner.log
** keysigner-compute.log
** keysigner-compute.log
* dbs: various db logs
* dbs: various db logs
* zeus: various zeus logs
* zeus: various zeus logs


Line 156: Line 163:
* Server Source: https://github.com/mozilla/browserid
* Server Source: https://github.com/mozilla/browserid
* RP/Client source: https://github.com/lloyd/myfavoritebeer.org/
* RP/Client source: https://github.com/lloyd/myfavoritebeer.org/
* RP/Client source: https://github.com/lloyd/myfavoritebooze.org/
* RP/Client source: https://github.com/mozilla/123done
* RP/Client source: https://github.com/lloyd/myfavoriteshow.org/
* Primary source: https://github.com/lloyd/eyedee.me/
* Primary source: https://github.com/lloyd/eyedee.me/


Line 171: Line 177:
*** node bin/browserid
*** node bin/browserid
*** node bin/verifier
*** node bin/verifier
*** node bin/router
*** node bin/static
*** /usr/bin/node /opt/browserid/lib/verifier/verifier-compute.js
*** /usr/bin/node /opt/browserid/lib/verifier/verifier-compute.js
*** /usr/bin/node /opt/browserid/lib/bcrypt-compute.js
*** /usr/bin/node /opt/browserid/lib/bcrypt-compute.js
** secure webheads:
** secure webheads:
*** node bin/dbwriter
*** node bin/dbwriter
*** /usr/bin/node /opt/browserid/lib/bcrypt-compute.js
*** /usr/bin/node /opt/browserid/lib/bcrypt-compute.js
** keysigners:
** keysigners:
*** node bin/keysigner
*** node bin/keysigner
*** /usr/bin/node /opt/browserid/lib/keysigner/keysigner-compute.js
*** /usr/bin/node /opt/browserid/lib/keysigner/keysigner-compute.js
** databases:
** databases:
*** /bin/sh /usr/bin/mysqld_safe --datadir=/data/mysql --pid-file=/var/run/mysqld/mysqld.pid
*** /bin/sh /usr/bin/mysqld_safe --datadir=/data/mysql --pid-file=/var/run/mysqld/mysqld.pid
*** /usr/sbin/mysqld --basedir=/usr --datadir=/data/mysql --user=mysql --log-error=/data/mysql/db1.iddb.scl2.stage.svc.mozilla.com.err --open-files-limit=8192 --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
*** /usr/sbin/mysqld --basedir=/usr --datadir=/data/mysql --user=mysql --log-error=/data/mysql/db1.iddb.scl2.stage.svc.mozilla.com.err --open-files-limit=8192 --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
*** perl /usr/bin/pt-heartbeat --create-table -D browserid -u root --update
*** perl /usr/bin/pt-heartbeat --create-table -D browserid -u root --update
** Note: This assumes access to the main Stage Admin server. If you do not have SSH access, then get QA staff or Dev to verify the processes.
** Note: This assumes access to the main Stage Admin server. If you do not have SSH access, then get QA staff or Dev to verify the processes.


* Version Checks
* Version Checks
** Production: https://browserid.org/ver.txt
** Production: https://login.persona.org/ver.txt
** Dev: https://dev.diresworb.org/ver.txt
** Dev: https://login.dev.anosrep.org/ver.txt
** Stage: https://diresworb.org/ver.txt
** Stage: https://login.anosrep.org/ver.txt
** Note: Dev should always be running one version newer than Stage. Stage should be running one version newer than Production.
** Note: Dev should always be running one version newer than Stage. Stage should be running one version newer than Production.


Line 194: Line 206:
** See Stage Logging section above.
** See Stage Logging section above.
** Note: This assumes access to the main Stage Admin server. If you do not have SSH access, then get QA staff or Dev to verify that the logs are active.
** Note: This assumes access to the main Stage Admin server. If you do not have SSH access, then get QA staff or Dev to verify that the logs are active.
* Verify heartbeat on the servers
** Dev: https://dev.diresworb.org/__heartbeat__
** Prod: https://browserid.org/__heartbeat__
** Stage: https://diresworb.org/__heartbeat__
** all of them should return "ok" to the browser


=== Local Installs ===
=== Local Installs ===
Line 220: Line 226:
** https://wiki.mozilla.org/QA/BrowserID/BrowserID_Basic_Install#Running_Unit_Tests
** https://wiki.mozilla.org/QA/BrowserID/BrowserID_Basic_Install#Running_Unit_Tests
* Instructions are good for any installation...
* Instructions are good for any installation...
=== Automation Testing ===
* There are two main ways to run automation testing:
** Directly through the Selenium/Jenkins/Sauce Labs automation environment
** Locally using a stand-alone Selenium server and the built-in automation tests
* Selenium/Jenkins/Sauce Labs automation - see the following links
** https://wiki.mozilla.org/QA/BrowserID/BrowserID_Basic_Install#Running_the_Automation_Tests
** https://wiki.mozilla.org/Identity/QA#BrowserID_Automation
*** Selenium BID dashboard: BrowserID dashboard: http://qa-selenium.mv.mozilla.com:8080/
*** Browserid-tests repo: https://github.com/mozilla/browserid/tree/dev/automation-tests
*** Dave Hunt's BIDPOM (BrowserID Page-Object Model) repo: https://github.com/davehunt/bidpom
* Automation on the localhost - see the following links:
** https://wiki.mozilla.org/QA/BrowserID/BrowserID_Basic_Install#Running_the_Automation_Tests


=== Sanity/Acceptance/Smoke ===
=== Sanity/Acceptance/Smoke ===
Line 226: Line 247:
* See the Test Cases (listed in the "Links and Documentation" section above) for more information.
* See the Test Cases (listed in the "Links and Documentation" section above) for more information.


* Server-side (https://diresworb.org/)
* Server-side (https://login.anosrep.org/) [https://moztrap.mozilla.org/manage/cases/?filter-productversion=2&filter-suite=82 MozTrap] [https://github.com/mozilla/browserid/blob/dev/automation-tests/persona_server/tests/test_manage_account.py AUTO]
** Sign Up with a new account and one email
** [https://moztrap.mozilla.org/manage/case/989/ Sign Up with a new account and one email]
*** Verify correct email verification sent to correct account (email provider)
** [https://moztrap.mozilla.org/manage/case/990/ Sign In - Sign Out with a current account of one email]
** Sign In with a current account of one email
** [https://moztrap.mozilla.org/manage/case/992/ Change the password on a current account with one email]
** Sign Out from a current account of one email
** [https://moztrap.mozilla.org/manage/case/993/ Delete/Remove a current account with one email]
** Change the password on a current account with one email
*** Verify correct email verification sent to correct account (email provider)
** Delete/Remove a current account with one email


* Client-side (http://myfavoritebeer.org using one desktop OS with one browser)
* Client-side (http://myfavoritebeer.org or http://123done.org using one desktop OS with one browser)
** Sign In using one RP/client to create a new account with one email
** Sign In using one RP/client to create a new account with one email
*** Verify correct email verification sent to correct account (email provider)
*** Verify correct email verification sent to correct account (email provider)
Line 247: Line 265:
** Delete/Remove the account from the server-side
** Delete/Remove the account from the server-side


* Client-side (http://myfavoritebeer.org using Android with stock browser)
* Client-side (http://beta.myfavoritebeer.org or http://beta.123done.org using Android with stock browser)
** Sign In using one RP/client to create a new account with one email
** Sign In using one RP/client to create a new account with one email
*** Verify correct email verification sent to correct account (email provider)
*** Verify correct email verification sent to correct account (email provider)
Line 269: Line 287:
* Site Access - Stage
* Site Access - Stage
** Verify use of HSTS (for https:// only sites)
** Verify use of HSTS (for https:// only sites)
** Verify that the Stage server site can be accessed through secured HTTPS: https://www.diresworb.org/
** Verify that the Stage server site can be accessed through secured HTTPS: https://login.anosrep.org/
** Verify that the Stage server site can not be accessed using only HTTP, but instead gets redirected to the HTTPS site: http://www.diresworb.org/
** Verify that the Stage server site can not be accessed using only HTTP, but instead gets redirected to the HTTPS site: http://login.anosrep.org/
** Verify that the Stage RP/client site can be accessed through standard HTTP: http://beta.myfavoritebeer.org/
** Verify that the Stage RP/client site can be accessed through standard HTTP: http://beta.myfavoritebeer.org/ or http://beta.123done.org
** Verify that the Stage RP/client site gets a certificate warning/error when accessed through secured HTTPS: https://beta.myfavoritebeer.org/
** Verify that the Stage RP/client site gets a certificate warning/error or is blocked when accessed through secured HTTPS: https://beta.myfavoritebeer.org/, http://beta.123done.org
** Verify similar for the following RPs/clients:
** Verify similar for the following RPs/clients:
*** beta.myfavoritebooze.org
*** beta.myfavoriteshow.org
*** carrera.databits.net:9999/
*** carrera.databits.net:9999/


* Accounts and Email Verification
* Accounts and Email Verification
** Creating an account from the server site using Sign Up
** Creating an account from the server site using Sign Up
** Creating an account inline (at first use of an RP/client Sign In)
** Creating an account inline (at first use of an RP/client Sign In) [https://github.com/mozilla/BrowserID-Tests/blob/master/123done/tests/test_new_user.py AUTO]
** Email notifications for new accounts: verification email through email provider with proper email account listed, live verification link, etc.
** Email notifications for new accounts: verification email through email provider with proper email account listed, live verification link, etc. [https://github.com/mozilla/BrowserID-Tests/blob/master/123done/tests/test_new_user.py AUTO]
*** Test with emails/accounts on various, popular email servers/services
*** Test with emails/accounts on various, popular email servers/services
*** See more details in the next bulleted list
*** See more details in the next bulleted list
Line 289: Line 305:
* Email Notifications for server or RP/Client
* Email Notifications for server or RP/Client
** On the same OS
** On the same OS
*** Email notification and verification using one browser
*** Email notification and verification using one browser [https://github.com/mozilla/BrowserID-Tests/blob/master/123done/tests/test_new_user.py AUTO]
*** Email notification on one browser, verification on an another
*** Email notification on one browser, verification on an another
** Across multiple OS
** Across multiple OS
Line 310: Line 326:


* Accounts and Emails
* Accounts and Emails
** Adding additional emails to an account
** Adding additional emails to an account [https://github.com/mozilla/BrowserID-Tests/blob/master/123done/tests/test_add_another_email.py AUTO]
** Attempt to add an email (that may or may not be yours) from another account
** Attempt to add an email (that may or may not be yours) from another account
** Deleting one or more emails (without actually deleting the account)
** Deleting one or more emails (without actually deleting the account)
Line 402: Line 418:


=== Specific RP/Client Functionality ===
=== Specific RP/Client Functionality ===
* beta.myfavoritebeer.org
* http://beta.myfavoritebeer.org
** Verify that this site allows for standard authentication, sign in, log out, password changes, and email additions to a new or current account.
** Verify that this site allows for standard authentication, sign in, log out, password changes, and email additions to a new or current account.
** Verify different sign-in/verification paths for Primary emails vs. BID emails.
** Verify different sign-in/verification paths for Primary emails vs. BID emails.
** Regression: attempt RP Sign In use with and without third-party cookie support set in the browser  
** Regression: attempt RP Sign In use with and without third-party cookie support set in the browser  


* beta.myfavoritebooze.org: the "Always sign in using this email" feature
* http://123done.org
** Verify that the site allows the user to stay logged in across sessions, browser restarts, OS logout/login, and even OS/device restart.
** Add some specific test information here...
** Verify that the site always opens with the current account and email unless the user logs out of the site.
** Verify that the site always opens with the current account and email unless the browser preferences/settings are cleared.
** Verify different sign-in/verification paths for Primary emails vs. BID emails.
** Regression: attempt RP Sign In use with and without third-party cookie support set in the browser
 
* beta.myfavoriteshow.org: the required email feature
** Verify that the site allows the user sign-in/sign-up exactly once per account/email combination.
** Verify that the site allows the user to sign-in/sign-up with every email associated with an account.
** Verify that with each new browser session, the email field has been cleared, since this is a one-time sign-in/sign-up site.
** Verify with an email that is not currently associated with a BrowserID account.
** Verify with an email that is associated with a BrowserID account, but is not currently in use.
** Verify with an email that is associated with a BrowserID account, and is current in use on one of the other RPs/Clients.
** Verify different sign-in/verification paths for Primary emails vs. BID emails.
** Regression: attempt RP Sign In use with and without third-party cookie support set in the browser


=== UI ===
=== UI ===
Manual and automated testing on the RP/client and the BID server to cover all aspects of the current UI.
Manual and automated testing on the RP/client and the BID server to cover all aspects of the current UI.
* Stage: Account creation UI in https://www.diresworb.org/
* Stage: Account creation UI in https://www.diresworb.org/
* Stage: Account creation inline using http://beta.myfavoritebeer.org/ or other RP/client
* Stage: Account creation inline using http://beta.myfavoritebeer.org/ or http://123done.org
** Stage: Account creation inline using Primary support
** Stage: Account creation inline using Primary support


Line 459: Line 461:
* Confirm Email verification UI
* Confirm Email verification UI


* RP/Client-side UI after sign-in (beta.myfavoritebeer.org, beta.myfavoritebooze.org)
* RP/Client-side UI after sign-in (http://beta.myfavoritebeer.org, http://123done.org)
** There is a hello message of sorts: Yo, <NAME>!
** There is a hello message of sorts: Yo, <NAME>!
** A dummy logo
** A dummy logo
** A logout link
** A logout link
* And for beta.myfavoriteshow.org
** Simple string: "Yay! <current email address> has been added to our email list!"


* RP/Client links:
* RP/Client links:
* BrowserID: https://diresworb.org
* BrowserID: https://diresworb.org
* Source code: https://github.com/lloyd/myfavoritebeer.org/
* Source code: https://github.com/lloyd/myfavoritebeer.org/
* Source code: https://github.com/mozilla/123done
* Mozilla Labs: http://mozillalabs.com/
* Mozilla Labs: http://mozillalabs.com/
* bitzi link: http://www.flickr.com/photos/bitzi/236037776/
* unhosted dudes link: https://myfavouritesandwich.org/


* Server side UI:
* Server side UI:
Line 519: Line 518:


=== Localization ===
=== Localization ===
NEW SECTION - IN PROGRESS!
* Verify the UI flow with specific focus on localization, all text-based content
* Verify the UI flow with specific focus on localization, all text-based content
** Page content, layout, and strings
** Page content, layout, and strings
Line 526: Line 523:
** Email content, layout, and strings
** Email content, layout, and strings
** Messages, tool tips, errors
** Messages, tool tips, errors
*** Errors: will need to create/force error situations in order to verify localization of error strings - TBD
*** Errors: will need to create/force error situations in order to verify localization of error strings


* List of supported languages: TBD
* List of supported languages
** From a local install, check the following files:
*** config/l10n-prod.json (for Production)
*** config/l10n-all.json (for Dev, local installs, and Stage)
*** l10n-prod.json should always be a subset of l10n-all.json
** For Stage to Prod deployments, check the very latest pull request for config/l10n-prod.json
*** Example: https://github.com/mozilla/browserid/pull/1563
** Note: Focus will always be on languages that are fully translated
** Note: Focus will always be on languages that are fully translated
* REF: http://svn.mozilla.org/projects/l10n-misc/trunk/browserid/
* REF: http://svn.mozilla.org/projects/l10n-misc/trunk/browserid/
Line 582: Line 585:
** Server: https://github.com/mozilla/browserid/issues
** Server: https://github.com/mozilla/browserid/issues
** RPs/Clients: https://github.com/lloyd/myfavoritebeer.org/issues
** RPs/Clients: https://github.com/lloyd/myfavoritebeer.org/issues
** and https://github.com/mozilla/123done/issues


* Ideally, this regression would be in the form of a quick automated test/script to check key fixes/changes/updates.
* Ideally, this regression would be in the form of a quick automated test/script to check key fixes/changes/updates.
Line 698: Line 702:
* Server/service scenarios - failover/recover, temp outage, traffic flow, etc.
* Server/service scenarios - failover/recover, temp outage, traffic flow, etc.
** Zeus/load balancer issues
** Zeus/load balancer issues
** webhead for browserid and verifier
** webhead for browserid, verifier, router, and static
** swebhead for dbwriter
** swebhead for dbwriter
** keysigner for keysigner
** keysigner for keysigner
** mysql/db
** mysql/db


=== OPs Load Testing ===
=== Load Testing ===
* Load Testing in Stage is handled by OPs using BrowserID code provided by Dev.
* Load Testing in Stage is handled by QA using BrowserID code provided by Dev.
* QA should monitor all load tests run by OPs
* QA should monitor all load tests
* QA should also schedule specific load tests to cover various requirements for server-side testing.
* QA should also schedule specific load tests to cover various requirements for server-side testing.
** Work with OPs to queue up specific load tests to check Server stability, various metrics
** Work with OPs to queue up specific load tests to check Server stability, various metrics
* Monitor load tests with OPs dashboards
* Monitor load tests with OPs dashboards - e.g., http://pencil.scl2.stage.svc.mozilla.com/dash/stage/bid_webhead (requires access to the MPT VPN)
* Analyze text and visual data for functionality and peformance
* Analyze text and visual data for functionality and peformance


* REF:
* REF:
** load_gen: https://github.com/mozilla/browserid/blob/dev/bin/load_gen
** load_gen: https://github.com/mozilla/browserid/blob/dev/bin/load_gen
<pre>
An example command line to run the load test:
    nohup bin/load_gen -s https://verifier.login.anosrep.org -o -m 250000 -u 1/250
-s  -- the URL to the browserid server
-o  -- don't request static resources
-m  -- maximum active users to simulate (ADU)
-u  -- use this range of #@loadtest.domain users (these users already exist in the
      stage database. If running your own instance, do `CREATE_TEST_USERS=1000
      BROWSERID_FAKE_VERIFICATION=1 NODE_ENV=test_mysql npm start`).
I typically run the above command piped into this: 
    | /usr/local/bin/tai64n | /usr/local/bin/tai64nlocal \
      >> /path/to/load_gen.log-`date +%Y%m%d-%H%M` 2>&1 &


== QA Sign-Off ==
which will timestamp the output and save it in a logfile, etc.
</pre>
 
== QA Sign-Off for Stage ==
* Requirements and procedures:
* Requirements and procedures:
** Complete all required testing for the current weekly train: resolved/closed issues, suggested areas for QA focus, specific features and areas of test coverage, automation, etc.
** Complete all required testing for the current weekly train: resolved/closed issues, suggested areas for QA focus, specific features and areas of test coverage, automation, etc.
Line 721: Line 744:
** Add additional comments (without a sign-off) if this weekly release fails QA
** Add additional comments (without a sign-off) if this weekly release fails QA


== Major Test Areas: In Progress ==
== QA Testing for Production Pushes ==
=== Security and Privacy Issues ===
This is a work in progress!...
Planned - Work with Dev and PM team to come up with use cases and a way to test them.
 
* Verify push to PHX1 (/etc/hosts hack)
** Standard sanity testing across platforms, OS, and browsers
* Verify push to SCL2 (/etc/hosts hack)
** Standard sanity testing across platforms, OS, and browsers
* Locale verification
** Based on final pull request for config/l10n-prod.json
** Quick sanity test of new/changed/removed locales
* Check the verifier service before OPs undrains a datacenter during a push (for PHX1 and SCL2)
** REF: Bug 753728 - browserid.org/verify is on occasion throwing 500 errors (being caught on the load balancer)
** REF: Bug 753828 - monitor the state of zeus pools


=== Certification/Session Issues ===
== QA Sign-Off for Production ==
Planned - Work with Dev and PM team to come up with use cases and a way to test them.
* TBD: Requirements and procedures:
* Similar to sign-off for Stage, but applies to both data centers
** Add the Services Maintenance Window information to the deployment ticket.
** Complete all required testing for the push to Production.
** Update the deployment ticket with a summary of the completed testing.
** Add a sign-off to the deployment ticket if this Production push passes QA.
** Add additional comments (without a sign-off) if this Production push fails QA.


=== BrowserID Add-On ===
Planned - Work with Dev on testing once it becomes available. See the following:
* https://addons.mozilla.org/en-US/firefox/addon/browser-sign-in/
* http://www.shanetomlinson.com/2011/mozilla-session-api-tutorial/
* https://github.com/mozilla/browserid_addon/issues
* https://github.com/mozilla/browserid_addon


Back To ==> [[QA/Browser_Technologies/Services | Mozilla BT Services]]
Back To ==> [[QA/Browser_Technologies/Services | Mozilla BT Services]]
Edwong
Confirmed users
964

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/428222...672405"