Confirmed users
2,177
edits
TestEngineering/Performance/Triage Process: Difference between revisions
TestEngineering/Performance/Triage Process (view source)
Revision as of 16:10, 24 July 2023
, 24 July 2023Redirected page to Performance/Tools/Testing/Triage
(→How to handle intermittent bugs which cover a crash of Firefox?: Added missing malloc handling) |
(Redirected page to Performance/Tools/Testing/Triage) |
||
| (14 intermediate revisions by 4 users not shown) | |||
| Line 1: | Line 1: | ||
#REDIRECT [[Performance/Tools/Testing/Triage]] | |||
== Triage Workflow == | == Triage Workflow == | ||
=== Triage Duty === | |||
Your main goal during triage duty is to make sure bugs are labelled appropriately and quickly based on recent bug activity. This might mean checking bug activity once a day, perhaps doing some minimal investigation, and then updating the bug's priority, severity, product, status, need-info, etc. | |||
See [[#Queries|Useful Queries]]. | |||
* Triage incoming bugs as early as possible or at least once a day. | |||
* Intermittent failures: | |||
** Only investigate an intermittent failure if it has happened more than once. | |||
** Glimpse over the failure details, and if incomplete information has been added as the first comment, add the relevant part of the log as a new comment. ** If it’s a duplicate bug mark it as such, or if not related to the component move it immediately to the correct one. | |||
** Intermittent failures should have a priority of P5 by default, unless they need investigation and a fix immediately. Then set a priority of P2 and find an owner. | |||
** On Monday the triage owner or person on triage duty goes through all the bugs that got updated by the intermittent failures bot. If there is a top-occurring failure make sure to assign the bug to someone familiar with the affected code. Failures which happened less often (like less than 15 times in the last week) you can simply ignore. | |||
* Untriaged bugs: | |||
** Bugs without a priority set should move to P3 by default, which means it will be fixed at some point. Only set P2 if the bug blocks current OKRs. | |||
* Mentored bugs: | |||
** It's generally up to the bug mentor to keep these bugs in good shape. Feel free to need-info the mentor if you have any doubts. | |||
** Set needinfo on the most recent contributor if they haven't replied for more than a week. | |||
** Never set a contributor as assignee. This will be done automatically by Phabricator when the initial patch gets submitted. Reset the assignee and set the bug to new if no further response comes in within a week. | |||
** Leave the priority as is and don't change it to P1 if such a bug gets assigned. | |||
* If it is not clear how to proceed on the bug, or if further input is necessary from stakeholders, add the whiteboard entry '''[perftest:triage]'''. Those bugs will be discussed in the next [https://docs.google.com/document/d/1SeMijarFsdtm-mrxkIQzV4y1PHcJN72JDPWOlxJ7u-A/edit#heading=h.v37yirv4o0rn triage meeting]. | |||
=== Review Queue === | |||
To ensure that we're responding to review requests in a timely manner, the [https://phabricator.services.mozilla.com/tag/perftest-reviewers/ #perftest-reviewers] group is triaged once/day. This involves tracking the number of open review requests and assigning a team member to be responsible for the review. See [[../Review Process/]] for more information about how reviews should be performed. | |||
* Open the [https://phabricator.services.mozilla.com/dashboard/view/84/ FxPerfTest dashboard] in Phabricator. | |||
* For any review that '''only''' has the '''#perftest-reviewers''' as the reviewer, assign a team member as a blocking reviewer. | |||
** The reviewer should be the next team member in rotation to balance the load across the team, however this may not be desirable if many large reviews are building up on an individual. Use the team member tabs to understand the review queue for individuals and your best judgement. | |||
=== Queries === | |||
* [https://mozilla.github.io/triage-center/?component=Testing%3AAWSY&component=Testing%3APerformance&component=Testing%3ARaptor&component=Testing%3ATalos Triage Center] highlights where attention is needed. | |||
* [https://bugzilla.mozilla.org/buglist.cgi?priority=--&keywords=meta%2C%20&query_format=advanced&product=Testing&resolution=---&list_id=15153821&keywords_type=nowords&component=AWSY&component=Performance&component=Raptor&component=Talos Untriaged bugs] | |||
* [https://bugzilla.mozilla.org/buglist.cgi?keywords_type=allwords&resolution=---&component=AWSY&component=Performance&component=Raptor&component=Talos&product=Testing&keywords=intermittent-failure&query_format=advanced&list_id=15222319 Intermittent failures] | |||
* [https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&o1=isnotempty&component=AWSY&component=Performance&component=Raptor&component=Talos&f1=bug_mentor&resolution=---&product=Testing&list_id=15153825 Mentored bugs] | |||
=== Triage Duty versus Triage Owner === | |||
* Every bug component has a Triage Owner. This is an ongoing, long-term role. | |||
* Anyone on the team may be assigned to Triage Duty. This is a short-term role that involves monitoring incoming bugs on a daily basis. | |||
* The triage team decides who is on triage duty until the next triage meeting, which means triage duty usually rotates on a weekly basis. | * The triage team decides who is on triage duty until the next triage meeting, which means triage duty usually rotates on a weekly basis. | ||
=== Bugs being worked on === | === Bugs being worked on === | ||
| Line 24: | Line 60: | ||
* P5 - Used for intermittent failures, or no intention to fix but will accept patches | * P5 - Used for intermittent failures, or no intention to fix but will accept patches | ||
== | == Strategies for triaging intermittents == | ||
* Look for patterns in Treeherder's [https://treeherder.mozilla.org/intermittent-failures.html intermittent failures view] (platforms, build types, tree etc). This also linked to in the Orange Factor field on each bug. | |||
** E.g. Investigate the intermittent logs associations with a grain of salt, as Code sheriffs may occasionally misattribute some failure logs. | |||
** E.g. if 90% of failures happen on Android, and the rest on some desktop platforms, there’s a chance that desktop failures were incorrectly assigned. | |||
* Recognise and mark duplicates as early as possible ([https://bugzilla.mozilla.org/show_bug.cgi?id=1552812 example]) | |||
* Use generic intermittent bugs when availabe ([https://bugzilla.mozilla.org/show_bug.cgi?id=1609295 example]) | |||
** Simply ask a Code sheriff to group them (a needinfo? + some guidelines should suffice) | |||
** Use this when you have lots of bugs covering the exact same underlying issue | |||
*** Pick the oldest bug | |||
*** Replace parts of the bug summary with <random> | |||
*** Use this only for common patterns you notice | |||
*** There are some risks involved here, especially if we’re not entirely sure about the underlying problem. Any mistake could hide other Raptor regressions. | |||
*** Making this too generic can increase the failure rate for what seems to be a common culprit. Code sheriffs will then have more reasons to turn our tests off. | |||
=== How to handle intermittent bugs which cover a crash of Firefox? === | === How to handle intermittent bugs which cover a crash of Firefox? === | ||
First make sure it doesn't stay in the Testing component but gets moved to a product that covers Firefox. Therefore check the crashing thread and find the reported crash frame as listed in the summary of the bug. From there go on and find the first frame that is part of our code. Also check for the following: | First make sure it doesn't stay in the Testing component but gets moved to a product that covers Firefox as crashes relate to problems to Firefox itself & not to the test harness. Therefore check the crashing thread and find the reported crash frame as listed in the summary of the bug. From there go on and find the first frame that is part of our code. Also check for the following: | ||
* For a header file (.h) you can most likely continue to the next frame | * For a header file (.h) you can most likely continue to the next frame | ||
| Line 34: | Line 84: | ||
* For allocation issues (like OOM) also find the appropriate caller | * For allocation issues (like OOM) also find the appropriate caller | ||
If not done yet, also add a comment with the link to the exact crash location. Make sure to keep the changeset id in the URL. | |||
* Figure out the right component | |||
** Don’t rush & assume that the 1st frame of the crashing thread is the culprit, especially if its corresponding source code points to a header (*.h) file | |||
** If indeed 1st frame isn’t the culprit, just go to the next frame from the logs. | |||
** Note: most often, this is not a trivial task. So even if you end up to another source file, it’s still very likely that the problem happens a bit more up the stack. If you get blocked, request an engineer’s assistance and learn from their process. | |||
* How do you know which engineer to ask for assistance? | |||
** By looking over the source file (you got stuck at) & figuring out its component (use Mercurial’s blame feature). With that component, you identify the team that likely has more knowledge over the problem. Contact the team & ask someone there to assist you. | |||
** You can also search for the associated file name in [https://searchfox.org/ searchfox], and find the corresponding Bugzilla component within the nearest <code>moz.build</code> file. | |||
== FAQ == | |||
=== Can I have many bugs assigned to me? === | === Can I have many bugs assigned to me? === | ||