MozillaWiki

QA/Execution/Web Testing/roles/buildmaster: Difference between revisions

Page Discussion

QA/Execution/Web Testing/roles/buildmaster (view source)

Revision as of 17:49, 21 July 2016

158 bytes removed ,  21 July 2016
→‎Co-owner
 
(26 intermediate revisions by 3 users not shown)
Line 1: Line 1:
=Introduction=
=Introduction=
Create a set of guidelines for investigating failures in our public Jenkins instance. This would be used by whoever is monitoring for failures, but would also be valuable for community who want to dive in. It should include how to identify failures, how to determine if they're already known, how to replicate them locally, how to determine if they're application bugs or test bugs, where to raise them, who to notify, and even how to fix them (if they're test failures) and submit pull requests. This could form part of a boot camp similar to other teams. -- from our 2015, [https://etherpad.mozilla.org/webqa-goals-2015-q2 Q2 goals brainstorm].
The buildmaster role is assigned to a team member in rotation, each lasting two weeks. The role is intended to identify failures early, and to take action towards resolving the failures. This action will vary depending on the skills and knowledge of the current buildmaster, or the level of commitment the team has to the affected project.


=Open Questions=
=Monitoring Failures=
* In the interests of reducing complexity, do we need tiers? Are we anticipating that there will be so many failures that some need prioritizing over others?
Failures can be monitored in the following ways:
* On the topic of sending daily emails - I think this would be extra work and considered noise by most recipients. It could be done via a whiteboard entry and Bugzilla whines.
* [https://webqa-ci.mozilla.com/ Jenkins dashboard]
* On the topic of checking builds once a day - There are several methods for doing this: view the web dashboard, subscribe to RSS feeds, read e-mail alerts, watch IRC notifications. We could consider others using Jenkins plugins.
* [https://webqa-ci.mozilla.com/rssFailed RSS feed]
* [[QA/Execution/Web_Testing/Automation/Build_Notifications|Mailing list]]
* [http://widget.mibbit.com/?settings=0a187437b06ca9bcb0cff2a717e985c5&server=irc.mozilla.org&channel=mozwebqa&noServerTab=false&autoConnect=true IRC]
 
=Roles=
==Owner==
* in Jenkins, on the failing job's output, you'll see "This build was not claimed. Claim it." Click on Claim it and add your relevant details.
* you'll now be the Assignee, and you and your investigation details should now show up next to the failing build, on https://webqa-ci.mozilla.com/claims/
* if you know or suspect the failure's root cause does or will affect other jobs/projects, send an email to either the public (preferred) or internal mailing list, as appropriate
* once/when the build passes, your claim and the failing job will automatically be removed from the [https://webqa-ci.mozilla.com/claims/ Claims view]
 
==Co-owner==
* serves as a "backup" in case of vacation/PTO
* both Owner and Co-owner should coordinate to help ensure continued coverage
* prompt the Owner with an offer for help, when there doesn't appear to be traction/visibility


=Rotation=
=Rotation=
{{WebQABuildMasterRotation}}
{{:QA/Execution/Web Testing/roles/buildmaster/Rotation}}
 
=Definition=
* buildmaster role last 2 weeks
* edit jenkins desc with your name as buildmaster
* check builds at least once per day (there are several methods for doing this: view the web dashboard, subscribe to RSS feeds, read e-mail alerts, watch IRC notifications. We could consider others using Jenkins plugins)
** investigate failures
** If it is a locator issue, if you have a question, if you wonder if the test is still valid or important then file a GitHub issue
** File bugs on projects based on info below, then contact the noted team members
* File Git Issues for test failures that require a test update.
** Label issue "test failure" with a priority
** Xfail the failing test to get the build green
* buildmaster is point of contact for open issues/bugs
* role includes filing bugs/issues, sending out emails, investigating issues
* does NOT include escalation paths, prioritizing fixes, following up with other teams
* send daily email with list of generally prioritized github issues to be fixed, blocking bugs that were filed


=Known Issues=
=Known Issues=
For the latest known issues check [https://etherpad.mozilla.org/webqa-buildmaster this etherpad]. If you are the current buildmaster please try to keep this pad updated. It helps when it comes to hand the role onto the next buildmaster, and can avoid duplicating effort investigating failures.
For the latest known issues check [https://public.etherpad-mozilla.org/p/webqa-buildmaster this etherpad]. If you are the current buildmaster please try to keep this pad updated. It helps when it comes to hand the role onto the next buildmaster, and can avoid duplicating effort investigating failures.


=Support Tiers=
=New Failures=
 
When a new failure is found, details should be added to [https://public.etherpad-mozilla.org/p/webqa-buildmaster this etherpad] and an issue should be raised in the appropriate place (see projects section for details). This is the minimum expected of the buildmaster, however if you have the time and skills (or time to learn the skills), it's a good idea to try to replicate the failure locally. If the failure is clearly something that should be fixed in the test suite, then it's also appreciated if you can submit a pull request with a fix.
===Tier 1===
* Marketplace
* AMO
* Mozilla.org
 
===Tier 2===
* SUMO
* Socorro
 
===Tier 3===
* BIDPOM
* Moztrap
* One and Done
 
===Tier 4 (Unsupported)===
* Affiliates
* Mozillians


=Projects=
=Projects=


===amo===
===Add-ons (AMO)===
* high priority
* Priority: High
* fails: 1st krupa / 2nd mathieu agopian FRANCE / 3rd wil clouser
* IRC: [irc://irc.mozilla.org/#remora #remora]
* file a bug
* Bugs: [https://bugzilla.mozilla.org/buglist.cgi?product=addons.mozilla.org&resolution=--- Open Bugs], [https://bugzilla.mozilla.org/enter_bug.cgi?product=addons.mozilla.org Raise Bug]
** https://bugzilla.mozilla.org/enter_bug.cgi?product=addons.mozilla.org
* Contacts:
*# Krupa Raj [:krupa]
*# Mathieu Agopian [:magopian]
*# Wil Clouser [:clouserw]


===bidpom===
===bidpom===
* towards low priority
* towards low priority
* fail: john morrison [jrgm] if infrastructure related (time outs, buttons not loading, etc.); bob or davehunt are the ones to fix
* fail: john morrison [jrgm] if infrastructure related (time outs, buttons not loading, etc.); bob or davehunt are the ones to fix
* IRC: mozwebqa
* IRC: [irc://irc.mozilla.org/#identity #identity]
* known bug file bug and also need info him; esp if you know who checked in the change who made it fail
* known bug file bug and also need info him; esp if you know who checked in the change who made it fail
** https://github.com/mozilla/bidpom/issues
** https://github.com/mozilla/bidpom/issues
===Bouncer===
* IRC: [irc://irc.mozilla.org/#bouncer #bouncer]
* Bugs: [https://bugzilla.mozilla.org/buglist.cgi?product=Webtools&component=Bouncer&resolution=---&list_id=12662997 Open Bugs], [https://bugzilla.mozilla.org/enter_bug.cgi?product=Webtools&component=Bouncer Raise Bug]
* Contacts:
*# Jeremy Orem [:oremj]


===Hello (Loop)===
===Hello (Loop)===
* IRC: #loop
* IRC: [irc://irc.mozilla.org/#loop #loop]
* Bugs: [https://bugzilla.mozilla.org/buglist.cgi?product=Loop&component=Client&resolution=---&list_id=12223283 Open Bugs], [https://bugzilla.mozilla.org/enter_bug.cgi?product=Loop&component=Client Raise Bug]  
* Bugs: [https://bugzilla.mozilla.org/buglist.cgi?product=Loop&component=Client&resolution=---&list_id=12223283 Open Bugs], [https://bugzilla.mozilla.org/enter_bug.cgi?product=Loop&component=Client Raise Bug]  
* Releases: https://github.com/mozilla/loop-client
* Releases: https://github.com/mozilla/loop-client
* Contacts: Mark Banner (Standard8), Dan Mosedale (dmose)
* Contacts:
 
*# Mark Banner [:Standard8]
===Marketplace===
*# Dan Mosedale [:dmose]
* high priority
* fails: krupa 1st / 2nd david durst EST/ wil clouser 3rd
* IRC: #marketplace
* check
** https://github.com/mozilla/fireplace
** https://github.com/mozilla/zamboni
* file a bug - if we know there's been a regression, if you can reproduce manually and you can correlate with what you see in automation, or if it's infrastructure related
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Marketplace
 
===mozilla.org===
* medium-high priority
* IRC: #www
* Contacts: Josh Mize (jgmize), Paul McLanahan (pmac)
* Repository: https://github.com/mozilla/bedrock
* Bugs: [https://bugzilla.mozilla.org/buglist.cgi?product=www.mozilla.org&component=General&resolution=---&list_id=12230511 Open Bugs], [https://bugzilla.mozilla.org/enter_bug.cgi?product=www.mozilla.org Raise Bug]


===mozillians.org===
===mozillians.org===
* medium
* medium
* IRC: #commtools
* IRC: [irc://irc.mozilla.org/#commtools #commtools]
* Contacts: nemo-yiannis
* Contacts: nemo-yiannis
* Bugs: [https://bugzilla.mozilla.org/buglist.cgi?order=Importance&list_id=12247087&resolution=---&classification=Other&query_format=advanced&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&component=Phonebook&product=Participation%20Infrastructure Open Bugs], [https://bugzilla.mozilla.org/enter_bug.cgi?product=Participation%20Infrastructure&component=Phonebook Raise Bug]
* Bugs: [https://bugzilla.mozilla.org/buglist.cgi?order=Importance&list_id=12247087&resolution=---&classification=Other&query_format=advanced&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&component=Phonebook&product=Participation%20Infrastructure Open Bugs], [https://bugzilla.mozilla.org/enter_bug.cgi?product=Participation%20Infrastructure&component=Phonebook Raise Bug]
Line 94: Line 70:
===MozTrap===
===MozTrap===
* lower priority
* lower priority
* IRC: #moztrap
* IRC: [irc://irc.mozilla.org/#moztrap #moztrap]
* Bugs: [https://bugzilla.mozilla.org/buglist.cgi?product=Mozilla%20QA&component=MozTrap&resolution=---&list_id=12235937 Open Bugs], [https://bugzilla.mozilla.org/enter_bug.cgi?component=MozTrap&product=Mozilla%20QA Raise Bug]  
* Bugs: [https://bugzilla.mozilla.org/buglist.cgi?product=Mozilla%20QA&component=MozTrap&resolution=---&list_id=12235937 Open Bugs], [https://bugzilla.mozilla.org/enter_bug.cgi?component=MozTrap&product=Mozilla%20QA Raise Bug]  
* Releases: https://github.com/mozilla/loop-client
* Contacts:
* Contacts: Peter Bengtsson (peterbe), Cameron Dawson (camd)
*# Peter Bengtsson [:peterbe]
 
*# Cameron Dawson [:camd]
* file a bug
*** https://bugzilla.mozilla.org/enter_bug.cgi?


===mozwebqa dashboard===
===mozwebqa dashboard===
* low priority
* low priority
* fail: bob
* fail: bsilverberg / (justinpotts?)
* IRC: mozwebqa
* IRC: mozwebqa
* https://github.com/mozilla/mozwebqa-dashboard/issues
* https://github.com/mozilla/mozwebqa-dashboard/issues
Line 110: Line 84:
===One and Done===
===One and Done===
* low priority
* low priority
* bob
* bsilverberg
* https://bugzilla.mozilla.org/enter_bug.cgi?component=One%20and%20Done&product=Mozilla%20QA
* https://bugzilla.mozilla.org/enter_bug.cgi?component=One%20and%20Done&product=Mozilla%20QA


===Socorro===
===Crash Stats (Socorro)===
* low priority
* Priority: Low
* tests will be pared down to match less upcoming features
* IRC: [irc://irc.mozilla.org/#breakpad #breakpad]
* fails:
* Bugs: [https://bugzilla.mozilla.org/buglist.cgi?product=Socorro&resolution=--- Open Bugs], [https://bugzilla.mozilla.org/enter_bug.cgi?product=Socorro Raise Bug]
** contact: mbrandt and rhelmer in #breakpad
* Contacts:
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Socorro
*# Matt Brandt [:mbrandt]
*# Peter Bengtsson [:peterbe]


===Sumo===
===Snippets===
* low priority
* IRC: [[irc://irc.mozilla.org/#snippets #snippets]
* fails: Mythmon, rdalal, r1cky [who is leaving SUMO]
* Bugs: [https://bugzilla.mozilla.org/buglist.cgi?product=Snippets&resolution=---&list_id=12663032 Open Bugs], [https://bugzilla.mozilla.org/enter_bug.cgi?product=Snippets Raise Bug]
* contact: IRC sumodev
* Contacts:
* known bug file bug and also need info him; esp if you know who checked in the change who made it fail
*# Giorgos Logiotatidis [:giorgos]
** https://bugzilla.mozilla.org/enter_bug.cgi?product=support.mozilla.org


===QMO===
===Support (SUMO)===
* low priority
* Priority: Low
* tests not in jenkins
* IRC: [irc://irc.mozilla.org/#sumodev #sumodev]
* could make a public job for contributors to workon or fix
* Bugs: [https://bugzilla.mozilla.org/buglist.cgi?product=support.mozilla.org&resolution=--- Open Bugs], [https://bugzilla.mozilla.org/enter_bug.cgi?product=support.mozilla.org Raise Bug]
* https://bugzilla.mozilla.org/enter_bug.cgi?product=quality.mozilla.org
* Contacts:
*# Rebecca Billings [:rbillings]
*# Michael Cooper [:mythmon]
*# Rehan Dalal [:rdalal]


=FAQ=
=FAQ=
Stephend
Confirmed users
9,511

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/1079926...1140600"