View source for Identity/Firefox Accounts/Testing Notes

= Notes =

(jrgm asks - “where do the notes below come from?”)

stuart: discussions with karlof and some devs, they are messy and incoherent ;) i’ll clean this up shortly

(jrgm asks - “does the FxA grouping include Oauth and Profile server?”)

stuart: it does yes, would it make more sense to break it up?

Sandbox env called ‘stable'
* dev are maintaining this 
* close to production

Ops is converting deployment to jenkins by end of Q1
* want stable env done the same way, will be done by ops as well

from dev -> staging, how to orchestrate automate version selecting for various project versions and deps?

front end code has functional tests

does load test look good, does api look good?

perhaps release one project a time, in order

content depends on auth 

almost like they are separate projects, released independently

TeamCity runs tests currently on site for ‘latest'
* ask jrgm or vlad

Able, an a/b testing service
* ship feature, turn on and off
* expand tests if needed
* john getting this on staging

future, 1% deployments, monitoring and testing : hard cutover currently

after hits stage john runs load testing

might make sense to get from dev -> prod quickly to get things like load or performance tested

can always talk to fxa-dev list to get some dev more ideas or chat directly with ryan kelly, vlad, or shane tomlinson

= Next Steps =
Make stable/latest test coverage as close as possible to dev/stage (or vice versa)

Ensure we have good test coverage for: 
* api (functional tests for requests/queries)
* basic e2e (login/logout, create account, etc.)
* load (how many rps?)
* performance (does response time vary by account? can a really junky/large account be slower?)

How can we orchestrate automated version selecting for various projects, taking into account potential version and dependency conflicts?

Are there any manual steps that qa does that can be automated (basic validations like config checking, or tagging, or trains)?

Thinking forward to when we have everything deployed nicely into production with % rollouts, what kind of tests could we run in production that would help us verify how things are running before increasing %? monitoring tests

[[Category:CSQA Testing Notes]]