TestEngineering/Performance/Triage Process: Difference between revisions

Jump to navigation Jump to search

TestEngineering/Performance/Triage Process (view source)

Revision as of 14:34, 29 April 2020

215 bytes added ,  29 April 2020
m
add links, reorg
m (add initial draft of strategies)
m (add links, reorg)
Line 26: Line 26:


* [https://bugzilla.mozilla.org/buglist.cgi?priority=--&keywords=meta%2C%20&query_format=advanced&product=Testing&resolution=---&list_id=15153821&keywords_type=nowords&component=AWSY&component=Performance&component=Raptor&component=Talos Untriaged bugs]
* [https://bugzilla.mozilla.org/buglist.cgi?priority=--&keywords=meta%2C%20&query_format=advanced&product=Testing&resolution=---&list_id=15153821&keywords_type=nowords&component=AWSY&component=Performance&component=Raptor&component=Talos Untriaged bugs]
* [https://bugzilla.mozilla.org/buglist.cgi?priority=--&keywords=meta%2C%20&query_format=advanced&product=Testing&resolution=---&list_id=15153821&keywords_type=nowords&component=AWSY&component=Performance&component=Raptor&component=Talos Intermittent failures]
* [https://bugzilla.mozilla.org/buglist.cgi?keywords_type=allwords&resolution=---&component=AWSY&component=Performance&component=Raptor&component=Talos&product=Testing&keywords=intermittent-failure&query_format=advanced&list_id=15222319 Intermittent failures]
* [https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&o1=isnotempty&component=AWSY&component=Performance&component=Raptor&component=Talos&f1=bug_mentor&resolution=---&product=Testing&list_id=15153825 Mentored bugs]
* [https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&o1=isnotempty&component=AWSY&component=Performance&component=Raptor&component=Talos&f1=bug_mentor&resolution=---&product=Testing&list_id=15153825 Mentored bugs]


Line 51: Line 51:
* P5 - Used for intermittent failures, or no intention to fix but will accept patches
* P5 - Used for intermittent failures, or no intention to fix but will accept patches


== Strategies for investigating intermittents ==
== Strategies for triaging intermittents ==


* Look for patterns in intermittent failures view (platforms, build types, tree etc)
 
* Look for patterns in Treeherder's [https://treeherder.mozilla.org/intermittent-failures.html intermittent failures view] (platforms, build types, tree etc). This also linked to in the Orange Factor field on each bug.
** E.g. Investigate the intermittent logs associations with a grain of salt, as Code sheriffs may occasionally misattribute some failure logs.
** E.g. Investigate the intermittent logs associations with a grain of salt, as Code sheriffs may occasionally misattribute some failure logs.
** E.g. if 90% of failures happen on Android, and the rest on some desktop platforms, there’s a chance that desktop failures were incorrectly assigned.
** E.g. if 90% of failures happen on Android, and the rest on some desktop platforms, there’s a chance that desktop failures were incorrectly assigned.
* Recognise and mark duplicates as early as possible (example)
* Recognise and mark duplicates as early as possible ([https://bugzilla.mozilla.org/show_bug.cgi?id=1552812 example])
* Use generic intermittent bugs when there’s the case
* Use generic intermittent bugs when availabe ([https://bugzilla.mozilla.org/show_bug.cgi?id=1609295 example])
** Simply ask a Code sheriff to group them (a needinfo? + some guidelines should suffice)
** Simply ask a Code sheriff to group them (a needinfo? + some guidelines should suffice)
** Use this when you have lots of bugs covering the exact same underlying issue
** Use this when you have lots of bugs covering the exact same underlying issue
Line 65: Line 66:
*** There are some risks involved here, especially if we’re not entirely sure about the underlying problem. Any mistake could hide other Raptor regressions.
*** There are some risks involved here, especially if we’re not entirely sure about the underlying problem. Any mistake could hide other Raptor regressions.
*** Making this too generic can increase the failure rate for what seems to be a common culprit. Code sheriffs will then have more reasons to turn our tests off.
*** Making this too generic can increase the failure rate for what seems to be a common culprit. Code sheriffs will then have more reasons to turn our tests off.
* Move anything related to crashes out of Raptor, as crashes relate to problems to Firefox itself & not to the test harness
** Figure out the right component
** Don’t rush & assume that the 1st frame of the crashing thread is the culprit, especially if its corresponding source code points to a header (*.h) file
** If indeed 1st frame isn’t the culprit, just go to the next frame from the logs.
** Note: most often, this is not a trivial task. So even if you end up to another source file, it’s still very likely that the problem happens a bit more up the stack. If you get blocked, request an engineer’s assistance. Follow his work until he identifies the right component, so you learn from his experience.
** How do you know which engineer to ask for assistance? By looking over the source file (you got stuck at) & figuring out its component (use Mercurial’s blame feature). With that component, you identify the team that likely has more knowledge over the problem. Contact it & ask someone there to assist you.
== FAQ ==


=== How to handle intermittent bugs which cover a crash of Firefox? ===
=== How to handle intermittent bugs which cover a crash of Firefox? ===


First make sure it doesn't stay in the Testing component but gets moved to a product that covers Firefox. Therefore check the crashing thread and find the reported crash frame as listed in the summary of the bug. From there go on and find the first frame that is part of our code. Also check for the following:
First make sure it doesn't stay in the Testing component but gets moved to a product that covers Firefox as crashes relate to problems to Firefox itself & not to the test harness. Therefore check the crashing thread and find the reported crash frame as listed in the summary of the bug. From there go on and find the first frame that is part of our code. Also check for the following:


* For a header file (.h) you can most likely continue to the next frame
* For a header file (.h) you can most likely continue to the next frame
Line 82: Line 75:
* For allocation issues (like OOM) also find the appropriate caller
* For allocation issues (like OOM) also find the appropriate caller


When you found the final frame, search for the associated file name in searchfox, and find the corresponding Bugzilla component within the nearest moz.build file. If not done yet, also add a comment with the link to the exact crash location. Make sure to keep the changeset id in the URL.
If not done yet, also add a comment with the link to the exact crash location. Make sure to keep the changeset id in the URL.
 
* Figure out the right component
** Don’t rush & assume that the 1st frame of the crashing thread is the culprit, especially if its corresponding source code points to a header (*.h) file
** If indeed 1st frame isn’t the culprit, just go to the next frame from the logs.
** Note: most often, this is not a trivial task. So even if you end up to another source file, it’s still very likely that the problem happens a bit more up the stack. If you get blocked, request an engineer’s assistance. Follow his work until he identifies the right component, so you learn from his experience.
* How do you know which engineer to ask for assistance?
** By looking over the source file (you got stuck at) & figuring out its component (use Mercurial’s blame feature). With that component, you identify the team that likely has more knowledge over the problem. Contact it & ask someone there to assist you.
** You can also search for the associated file name in [https://searchfox.org/ searchfox], and find the corresponding Bugzilla component within the nearest <code>moz.build</code> file.
 
== FAQ ==


=== Can I have many bugs assigned to me? ===
=== Can I have many bugs assigned to me? ===
Mjzffr
Confirmed users
378

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/1226625"

Navigation menu