View source for User:Kthiessen/Scratch/Services QA Automation

'''------------------------------------------------------------------'''<br />
'''PROJECTS NOTES'''<br />
'''------------------------------------------------------------------'''

'''Kinto'''<br />
[http://ziade.org/kinto.html http://ziade.org/kinto.html]

'''------------------------------------------------------------------'''<br />
'''AUTOMATION'''<br />
'''------------------------------------------------------------------'''

'''RUN.SH'''<br />

* sourch a separate job name parser that sets env vars (for any job)
* add to _shared??
* {{bug|1209328}}
* Add cleanup script to NUKE all firefox running tasks
* <br />

'''PREFS.JS HANDLER'''

We want to store our prefs in json so they will be interoperable w/ Marionette<br />

* once job name is parsed, we need to set prefs indicated by &quot;env&quot; portion of job name
* mozprofile -p &quot;/Users/rpappalardo/Library/Application Support/Firefox/Profiles/tjli91ns.ddddd&quot; --preferences=./prefs/prefs.autopush.prod.json
* PRE-REQ: funciton to return full path given pathname
<br />
[https://ftp.mozilla.org/pub/utilities/profilemanager/1.0/ https://ftp.mozilla.org/pub/utilities/profilemanager/1.0/]<br />
[https://developer.mozilla.org/en-US/docs/Profile_Manager https://developer.mozilla.org/en-US/docs/Profile_Manager]

function path_profile(name_profile):<br />
 do the math<br />
 return path_profile

<br />
'''TEST PAGE'''<br />

* Add time stamp to see if screen shots change
* Add PST??
* <br />



'''REFRESH - SESSION'''<br />

* checkin TASKKILL to github (if not already)
* re-clone this to win server (this will eventually be re-clong at slave refresh time)
* that will fix these kinds of errors: [https://www.dropbox.com/s/vx1r68td9y9460r/Screenshot%202015-11-19%2011.42.33.png?dl=0 https://www.dropbox.com/s/vx1r68td9y9460r/Screenshot%202015-11-19%2011.42.33.png?dl=0]
* <br />
* <br />

'''------------------------------------------------------------------'''<br />
'''JENKINS'''<br />
'''------------------------------------------------------------------'''

'''REFRESH- SLAVES'''


* work on slave-refresh job
<br />
'''CONFIG.XML - UPDATES'''


complete Vagrant copy of encrypted configs to jenkins-dev
re-spin jenkins-dev with an IAM role
configure s3 plugin, test to make sure it works
# if so, re-encrypt config.xml and checkin
<br />
'''SLAVE RECONNECT ON FAIL'''

11:30:02 AM &lt;pdehaan&gt; a) get valid (non test) SSL cert.<br />
11:30:09 AM &lt;pdehaan&gt; b) get it hooked up in Jenkins<br />
11:30:16 AM &lt;pdehaan&gt; c) change port to 443 instead of 8443<br />
11:30:26 AM &lt;pdehaan&gt; d) install JNLP as a service

<br />
'''SITE CERT - JENKINS'''

This warrants its own etherpad:<br />
[https://public.etherpad-mozilla.org/p/services-qa-jenkins-cert https://public.etherpad-mozilla.org/p/services-qa-jenkins-cert]

'''JOB SCHEDULING'''<br />

* this will vary from job-to-job
* Our Jenkins can't store this info in a job (it must be puppetized)
* how to do this??
<br />
'''------------------------------------------------------------------'''<br />
'''EVIDENCE GATHERING'''<br />
'''------------------------------------------------------------------'''

SCREENSHOTS<br />

use s3 post-build step to copy screenshot artifacts to s3
link s3 files to appropriate s3 URL so link is visible in console logs
collect screenshots before and after every step
# on FAIL &gt; stick into ANIMGIF
# on PASS &gt; delete evidence
# console log: link to S3 bucket for that run


'''------------------------------------------------------------------'''<br />
'''BACKBURNER'''<br />
'''------------------------------------------------------------------'''<br />

* <br />

'''INTERMITTENT ERRORS'''<br />

screen resolution occasionally goes to infinity!!!
* maybe a kangaroo thing
pop notification not found - is this a legitimate push bug? can we gather more evidence
* more evidence added - circle back when more results


'''------------------------------------------------------------------'''<br />
'''SPRING CLEANING'''<br />
'''------------------------------------------------------------------'''

'''Remove old branches'''

 /feature/absearch-schema<br />
 /feature/e2e-slave<br />
 /feature/env-vars<br />
 /feature/hiera<br />
 /feature/https<br />
 /feature/job_configs<br />
 /feature/slaves<br />
 /feature/verify_job<br />
 /pdehaan/groovy-json

<br />
====

child projects here:<br />
[https://etherpad.mozilla.org/services-qa-jenkins https://etherpad.mozilla.org/services-qa-jenkins]<br />
[https://etherpad.mozilla.org/services-qa-dashboard https://etherpad.mozilla.org/services-qa-dashboard]



'''------------------------------------------------------------------'''<br />
'''TEST COVERAGE'''<br />
'''------------------------------------------------------------------'''

[1]. Bugzilla

- source of truth for actual test coverage

[2]. services-test

- branch = master: production tests<br />
 * tests that have been put through the paces<br />
 * tests that do verification + have discreet P/F output

- branch = dev: WIP tests<br />
 * WIP tests

<br />
------------------------------------------------------------------<br />
REPORTING<br />
------------------------------------------------------------------

OPTION #1: Dashboard (TBD)<br />
* see: [https://etherpad.mozilla.org/services-qa-dashboard https://etherpad.mozilla.org/services-qa-dashboard]<br />
* sphilp is doing this<br />
*<br />
OPTION #2: Treeherder<br />
[https://etherpad.mozilla.org/treeherder-kickstart https://etherpad.mozilla.org/treeherder-kickstart]<br />
From [https://etherpad.mozilla.org/services-qa-jenkins https://etherpad.mozilla.org/services-qa-jenkins]

<br />
------------------------------------------------------------------<br />
PRIORITIES<br />
------------------------------------------------------------------

Focus on proof-of-concept Docker containers next few weeks<br />
to determine:

[A] specifics of Docker utility as an all-around, portable testing tool + environment<br />
[B] can we drop using Jenkins altogether

see: DOCKER notes in REFERENCE (below)

<br />
TODO LIST

[Q3]

1. CONTAINER TYPE #1 - ubuntu desktop + VNC<br />
 - proof-of-concept: pocket application [sphilp]<br />
 - TODO: can everyone do this now?<br />
<br />
2. CONTAINER TYPE #2 - ENV var dependencies [kthiessen]<br />
 - get an image up on dockerhub (once secrets figured out, do we need to limit this to qa only on code commit? Okay for now, review in Q4<br />
 - proof-of-concept: absearch application (same as msisdn-gateway) [sphilp]<br />
 - get private DockerHub repo<br />
 - TODO: open external file, dynamically create env vars

3. CONTAINER TYPE #3 - multi-hosts + desktop [pdehaan]<br />
 - trello card - for loop - 2 desktops in docker container<br />
 - proof-of-concept: land Koki's Loop test [pdehaan]<br />
 - figure out multi-host container [rpapa]<br />
<br />
* env vars<br />
* multi-hosts from the same image

[Q4]

4. CONTAINER TYPE #4 - ad-hoc test execution (&amp; debugging)<br />
- a nice home for history and logs of runs<br />
- pipe stdout into a log dir?<br />
- sync that log dir to s3?<br />
- add a treeherder parser? - add Trello card

-----------------------------<br />
Q2 Highlevel Goals<br />
-----------------------------

People: sphilp, rpapa, kthiessen, pdehaan

Jenkins pipeline - sphilp<br />
- communication with ops jenkins - kthiessen<br />
- report results to ops jenkins - kthiessen<br />
- report results to treeherder as needed- rpapa<br />
- automate deploy tickets - rpapa<br />
- automate deploy verification - rpapa<br />
- automate deploy prod handoff - rpapa<br />
- types of tests run via jenkins jobs - DONE

e2e testing process - sphilp<br />
- where does code live - sphilp<br />
- how to output to treeherder - rpapa<br />
- how to get sherrif support - sphilp<br />
- how to get devs to maintain tests - sphilp

Treeherder output for e2e tests - sherrifed by a-team maintained by devs

Puppetize jenkins and jobs - sphilp

<br />
-----------------------------<br />
Weekly - 2015-04-10<br />
-----------------------------

Load test standards<br />
- what should we guarantee load at (we test all services up to n rps)<br />
* varies per service depending on what the service does<br />
* loadsv1 vs loadsv2 vs bees<br />
 * loadsv2 should centralized, collect metrics, find bottlenecks<br />
* repeated runs may introduce issues (reuse accounts or data, needs to be consistent every run)<br />
* collisions in simultaneous runs<br />
* only automation will be a guaranteed minimum of 10-100/rps (should be no problem for any api), basic 5 minute load test with say 10 concurrent<br />
* set guidelines per project, what scale, what load %

projects that need manual load testing<br />
* tiles - currently 9500 rps, likely increase to 11000<br />
* reading list - unknown<br />
* shavar - currently up to 4500 rps<br />
* fxa<br />
 * content - currently<br />
 * oauth - currently 1k rpm<br />
 * auth - currently 100k rpm<br />
* loop - currently ~120 rps. 80 concurrent users

may need better telemetry for user actions to get accurate requests numbers, heka numbers?

use production data to continually refine the load test parameters

what to look for, how to get useful information<br />
- jrgm can lead next week

Upcoming:<br />
Security Testing<br />
Failure Testing



------------------------------------------------------------------<br />
JRGM distinguished lecture series<br />
------------------------------------------------------------------

TOOLS<br />
- $ lsof is your friend. install it.<br />
 * shows web socket &amp; file activity as well as who's listening on what port<br />
 * brings together a lot of features of other tools<br />
 - $ top<br />
 * &quot;load average&quot; shows processes blocked -or- IO wait?<br />
 * 3 numbers - measure @ 1min, 5min, 15min (the last 2 being most useful)<br />
 * anything &gt; 1 shows that there is some processes queued (or IO waiting)<br />
[http://www.cyberciti.biz/tips/understanding-unixlinux-filesystem-inodes.html http://www.cyberciti.biz/tips/understanding-unixlinux-filesystem-inodes.html]<br />




------------------------------------------------------------------<br />
BEST PRACTICES<br />
------------------------------------------------------------------<br />
* [jenkins] update plugins - frequency: bi-weekly?

<br />
------------------------------------------------------------------<br />
REFERENCE<br />
------------------------------------------------------------------

child projects here:<br />
[https://github.com/mozilla-services/services-qa-jenkins https://github.com/mozilla-services/services-qa-jenkins]<br />
[https://github.com/mozilla-services/services-qa-dashboard https://github.com/mozilla-services/services-qa-dashboard]

[https://lumberjaph.net/provision-an-ec2-instance-with-vagrant-and-ansible/ https://lumberjaph.net/provision-an-ec2-instance-with-vagrant-and-ansible/]<br />
<br />
MOZTOOLS<br />
mozharness: talk to whimbo or maja_fz or releng channel<br />
[https://etherpad.mozilla.org/mozharness-overview https://etherpad.mozilla.org/mozharness-overview]<br />
<br />
DOCKER<br />
[https://mana.mozilla.org/wiki/display/SVCOPS/OPs+notes+for+converting+a+project+to+Docker https://mana.mozilla.org/wiki/display/SVCOPS/OPs+notes+for+converting+a+project+to+Docker]<br />
[https://mana.mozilla.org/wiki/display/SVCOPS/Docker+and+Cloud+Services https://mana.mozilla.org/wiki/display/SVCOPS/Docker+and+Cloud+Services]<br />




'''------------------------------------------------------------------'''<br />
'''TEST COVERAGE'''<br />
'''------------------------------------------------------------------'''

'''[1]. Bugzilla'''

- source of truth for actual test coverage

'''[2]. services-test'''

- branch = master: production tests<br />
 * tests that have been put through the paces<br />
 * tests that do verification + have discreet P/F output

- branch = dev: WIP tests<br />
 * WIP tests

<br />
'''------------------------------------------------------------------'''<br />
'''REPORTING'''<br />
'''------------------------------------------------------------------'''

'''OPTION #1: Dashboard (TBD)'''<br />
* see: [https://etherpad.mozilla.org/services-qa-dashboard https://etherpad.mozilla.org/services-qa-dashboard]<br />
* sphilp is doing this<br />
*<br />
'''OPTION #2: Treeherder'''<br />
[https://etherpad.mozilla.org/treeherder-kickstart https://etherpad.mozilla.org/treeherder-kickstart]<br />
From [https://etherpad.mozilla.org/services-qa-jenkins https://etherpad.mozilla.org/services-qa-jenkins]

<br />
'''------------------------------------------------------------------'''<br />
'''PRIORITIES'''<br />
'''------------------------------------------------------------------'''

Focus on proof-of-concept Docker containers next few weeks<br />
to determine:

[A] specifics of Docker utility as an all-around, portable testing tool + environment<br />
[B] can we drop using Jenkins altogether

see: DOCKER notes in REFERENCE (below)

<br />
'''TODO LIST'''

'''[Q3]'''

'''1. CONTAINER TYPE #1 - ubuntu desktop + VNC'''<br />
 - proof-of-concept: pocket application [sphilp]<br />
 - TODO: can everyone do this now?<br />
<br />
'''2. CONTAINER TYPE #2 - ENV var dependencies [kthiessen]'''<br />
 - get an image up on dockerhub (once secrets figured out, do we need to limit this to qa only on code commit? Okay for now, review in Q4<br />
 - proof-of-concept: absearch application (same as msisdn-gateway) [sphilp]<br />
 - get private DockerHub repo<br />
 - TODO: open external file, dynamically create env vars

'''3. CONTAINER TYPE #3 - multi-hosts + desktop [pdehaan]'''<br />
 - trello card - for loop - 2 desktops in docker container<br />
 - proof-of-concept: land Koki's Loop test [pdehaan]<br />
 - figure out multi-host container [rpapa]<br />
<br />
* env vars<br />
* multi-hosts from the same image

'''[Q4]'''

'''4. CONTAINER TYPE #4 - ad-hoc test execution (&amp; debugging)'''<br />
- a nice home for history and logs of runs<br />
- pipe stdout into a log dir?<br />
- sync that log dir to s3?<br />
- add a treeherder parser? - add Trello card

'''-----------------------------'''<br />
'''Q2 Highlevel Goals'''<br />
'''-----------------------------'''

People: sphilp, rpapa, kthiessen, pdehaan

Jenkins pipeline - sphilp<br />
- communication with ops jenkins - kthiessen<br />
- report results to ops jenkins - kthiessen<br />
- report results to treeherder as needed- rpapa<br />
- automate deploy tickets - rpapa<br />
- automate deploy verification - rpapa<br />
- automate deploy prod handoff - rpapa<br />
- types of tests run via jenkins jobs - DONE

e2e testing process - sphilp<br />
- where does code live - sphilp<br />
- how to output to treeherder - rpapa<br />
- how to get sherrif support - sphilp<br />
- how to get devs to maintain tests - sphilp

Treeherder output for e2e tests - sherrifed by a-team maintained by devs

Puppetize jenkins and jobs - sphilp

<br />
'''-----------------------------'''<br />
'''Weekly - 2015-04-10'''<br />
'''-----------------------------'''

Load test standards<br />
- what should we guarantee load at (we test all services up to n rps)<br />
* varies per service depending on what the service does<br />
* loadsv1 vs loadsv2 vs bees<br />
 * loadsv2 should centralized, collect metrics, find bottlenecks<br />
* repeated runs may introduce issues (reuse accounts or data, needs to be consistent every run)<br />
* collisions in simultaneous runs<br />
* only automation will be a guaranteed minimum of 10-100/rps (should be no problem for any api), basic 5 minute load test with say 10 concurrent<br />
* set guidelines per project, what scale, what load %

projects that need manual load testing<br />
* tiles - currently 9500 rps, likely increase to 11000<br />
* reading list - unknown<br />
* shavar - currently up to 4500 rps<br />
* fxa<br />
 * content - currently<br />
 * oauth - currently 1k rpm<br />
 * auth - currently 100k rpm<br />
* loop - currently ~120 rps. 80 concurrent users

may need better telemetry for user actions to get accurate requests numbers, heka numbers?

use production data to continually refine the load test parameters

what to look for, how to get useful information<br />
- jrgm can lead next week

Upcoming:<br />
Security Testing<br />
Failure Testing



'''------------------------------------------------------------------'''<br />
'''JRGM distinguished lecture series'''<br />
'''------------------------------------------------------------------'''

TOOLS<br />
- $ lsof is your friend. install it.<br />
 * shows web socket &amp; file activity as well as who's listening on what port<br />
 * brings together a lot of features of other tools<br />
 - $ top<br />
 * &quot;load average&quot; shows processes blocked -or- IO wait?<br />
 * 3 numbers - measure @ 1min, 5min, 15min (the last 2 being most useful)<br />
 * anything &gt; 1 shows that there is some processes queued (or IO waiting)<br />
[http://www.cyberciti.biz/tips/understanding-unixlinux-filesystem-inodes.html http://www.cyberciti.biz/tips/understanding-unixlinux-filesystem-inodes.html]<br />




'''------------------------------------------------------------------'''<br />
'''BEST PRACTICES'''<br />
'''------------------------------------------------------------------'''<br />
* [jenkins] update plugins - frequency: bi-weekly?

<br />
'''------------------------------------------------------------------'''<br />
'''REFERENCE'''<br />
'''------------------------------------------------------------------'''

child projects here:<br />
[https://github.com/mozilla-services/services-qa-jenkins https://github.com/mozilla-services/services-qa-jenkins]<br />
[https://github.com/mozilla-services/services-qa-dashboard https://github.com/mozilla-services/services-qa-dashboard]

[https://lumberjaph.net/provision-an-ec2-instance-with-vagrant-and-ansible/ https://lumberjaph.net/provision-an-ec2-instance-with-vagrant-and-ansible/]<br />
<br />
'''MOZTOOLS'''<br />
mozharness: talk to whimbo or maja_fz or releng channel<br />
[https://etherpad.mozilla.org/mozharness-overview https://etherpad.mozilla.org/mozharness-overview]<br />
<br />
'''DOCKER'''<br />
[https://mana.mozilla.org/wiki/display/SVCOPS/OPs+notes+for+converting+a+project+to+Docker https://mana.mozilla.org/wiki/display/SVCOPS/OPs+notes+for+converting+a+project+to+Docker]<br />
[https://mana.mozilla.org/wiki/display/SVCOPS/Docker+and+Cloud+Services https://mana.mozilla.org/wiki/display/SVCOPS/Docker+and+Cloud+Services]<br />


<br />