CrashKill: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
m (Reverted edits by Imansubarkah (talk) to last revision by Tomcat)
No edit summary
 
(284 intermediate revisions by 18 users not shown)
Line 1: Line 1:
__NOTOC__
==General Agenda==
==General Agenda==
Crashkill team works to track and escalate stability issues for both Desktop and Mobile products. Our activities include....
* Track general stability for all our releases and development channels.
* Triage Top Crashes to find owners, set priorities, and identify vender outreach issues.
* Triage Top Crashes to find owners, set priorities, and identify vender outreach issues.
* Ensure that crash bugs are logged and updated with new information.
* Identify top tools issues so we can find & fix faster.
* Identify top tools issues so we can find & fix faster.
* Identify the first thing we want to do to increase developer awareness and commitment to fixing top crashers.
* Identify the first thing we want to do to increase developer awareness and commitment to fixing top crashers.
* Identify the first thing we want to automate.
* Identify the first thing we want to automate.
* Working with the Socorro team to get features on the roadmap that help us find and diagnose crash bugs better.
* Set [[CrashKill/Topcrash|topcrash threshold criteria]].
==Top Crashes==
There are a number of reports that we triage regularly in order to stay on top of issues, most notably the topcrash reports as linked in the front page of Socorro crash-stats (see top links in the Reports & Tools section of this page). Finding reproducible steps and helping us make them actionable will have a direct and in many cases significant impact on our users due to the volume involved. Not all bugs are fixes in our code. Many are 3rd party issues that can be resolved through outreach and/or blocklisting.
== Reports & Tools ==
Mozilla collects crash data which can be easily viewed using Socorro (a web interface to the crash reports). We use Socorro and its numerous reports to monitor stability rates on a daily basis.
* https://crash-stats.mozilla.com/products/Firefox (Firefox Desktop)
* https://crash-stats.mozilla.com/products/FennecAndroid (Firefox for Android)
* https://crash-stats.mozilla.com/products/B2G (Firefox OS)
The [http://arewestableyet.com/ "Are We Stable Yet?" Stability Dashboard] tries to collect all important numbers in one place, with colored indicators for hinting at what they mean.


* Some end of 09q4 status summary  https://wiki.mozilla.org/CrashKill/Status


== Links ==
We also have a list of additional reports that we have developed to help us understand certain crashes and trends. We plan over time to integrate these into Socorro and most of these are on the roadmap already.


=== Bug lists ===
* https://crash-analysis.mozilla.com/chofmann/  
* [http://bit.ly/3K8mhq The CrashKill Bugs]  <small>(Status Whiteboard: [crashkill])</small>
* https://crash-analysis.mozilla.com/rkaiser/ (see [https://crash-analysis.mozilla.com/rkaiser/0000.overview.html Overview] for always-current links)
* {{Crash-ThirdParty| [http://bit.ly/1TlEts CrashKill Bugs caused by third parties] }} <small>(Status Whiteboard: [crashkill-thirdparty])</small>
** Explosiveness, Weekly bug stats, FxOS crashes by B2G version, Flash topcrashes by version, and other specialty reports
* {{Crash-Blocklist| [http://bit.ly/33K46O Crashkill Bugs where we're considering blocking the addon/dll] }}  <small>(Status Whiteboard: [crashkill-block])</small>
* https://crash-analysis.mozilla.com/bsmedberg/
* {{Crash-Fixed| [http://bit.ly/2aFggC CrashKill Bugs where we've provided a fix] of some kind, even if just a wallpaper or skidmark }}  <small>(Status Whiteboard: [crashkill-fix])</small>
** https://crash-analysis.mozilla.com/bsmedberg/flash-summary.html (summary breakdown of crashes and hangs by Flash version)
* {{Crash-Debug| [http://bit.ly/3qmv2U CrashKill Bugs where we've provide a patch only to help debug the problem] }} <small>(Status Whiteboard: [crashkill-debug])</small>
** [[CrashKill/Flash Breakdown Data]]
* Reporting Malware (external link): https://www.techsupportalert.com/content/how-report-malware-or-false-positives-multiple-antivirus-vendors.htm


* [http://bit.ly/Uf9J5 List] of bugs tracking metrics improvements for CrashKill efforts.  <small>(Status Whiteboard: [crashkill-metrics])</small>
== Planning ==
* nominate bugs for the 3.6-only DLL blocklist by adding a dependency to {{bug|525103}}
* [[CrashKill/Plan]]
* [http://bit.ly/6Cv0v7 SkipList Bugs] and [https://wiki.mozilla.org/Breakpad/SkipList how to make a SkipList] request
* We are actively working with the Socorro team on a list of [https://wiki.mozilla.org/CrashKill/Plan/Priorities priorities] for getting features into Socorro that will help crash analysis. We update the list quarterly.


=== Analysis ===
==Work Weeks==
* [http://crash-stats.mozilla.com/topcrasher/byversion/Firefox/3.5.5/7 Firefox 3.5.5 top crashes in the last week]
*[[CrashKill/WorkWeek2012|Stability Work Week 2012]] (Notes)
* [http://crash-stats.mozilla.com/topcrasher/byversion/Firefox/3.6b4/7 Firefox 3.6 beta 4 top crashes in the last week]
*[[CrashKill/StabilityWeek2013|Stability Week 2013]] (Notes)
* [http://people.mozilla.com/crash_analysis/ Correlation Reports]
* [[CrashKill/Signatures|Crash signatures by module]]
* [http://people.mozilla.com/~chofmann/crash-data/new-crashes/ chofmann's new crash signatures]
* [http://people.mozilla.org/~jst/new-crashes/Firefox/latest/ jst's new crash signatures, running daily]
* [http://bit.ly/4XsjUp Outreach Bugs] tracking our efforts to contact add on vendors who have products causing crashes that we want to kill.
* [https://wiki.mozilla.org/CrashKill/CrashDash crash bug trend dashboard] some metrics around crash bugs filed, fixed, and resolved through triage.


* [http://www.squarefree.com/2009/10/15/531/ How to sort a bug list in topcrash order]
==Weekly Crashkill Meetings==
* [[CrashKill/Mac Flash Identifiers|Mac Flash Identifiers]] - Turns "025105C956638D665850591768FB743D0" into "Flash 10.0.32.18 on PPC"
Up to the end of 2014, weekly meetings were held to talk about stability issues across releases and channels. Those now have been replaced by mailing list communication and ad-hoc meetings before and after releases.


==Meeting logistics==
[[CrashKill/Meetings|Archived info about meetings]]
* 650-903-0800 x92 Conf# 290 (US/INTL)
* 1-800-707-2533 (pin 369) Conf# 290 (US)
* *1 to unmute yourself
* join irc.mozilla.org #planning for back channel
* Mountain View conference room "Warp Core"


==Meeting notes==
==Old Stuff==
* [[CrashKill/2010-08-10|Tuesday, August 10, 12:30pm]]
* [[CrashKill/Historical| Old links, queries and info]]
* [[CrashKill/2010-05-04|Tuesday, May 4, 12:30pm]]
* [[CrashKill/2010-03-01|Monday, March 1, 3pm PST]]
* [[CrashKill/2010-02-08|Monday, February 8, 3pm PST]]
* [[CrashKill/2010-02-01|Monday, February 1, 3pm PST]]
* [[CrashKill/2009-12-14|Monday, December 14, 3pm PST]]
* [[CrashKill/2009-11-30|Monday, November 30, 3pm PST]]
* [[CrashKill/2009-11-23|Monday, November 23, 3pm PST]]
* [[CrashKill/2009-11-16|Monday, November 16, 3pm PST]]
* [[CrashKill/2009-11-09|Monday, November 9, 3pm PST]]
* [[CrashKill/2009-11-02|Monday, November 2, 3pm PST]]
* [[CrashKill/2009-10-26|Monday, October 26, 3pm PDT]]
* [[CrashKill/2009-10-19|Monday, October 19, 3pm PDT]]
* [[CrashKill/2009-10-12|Monday, October 12, 3pm PDT]]
* [[CrashKill/2009-10-05|Monday, October 5, 3pm PDT]]
* [[CrashKill/2009-09-28|Monday, September 28, 3pm PDT]]

Latest revision as of 20:10, 22 December 2014

General Agenda

Crashkill team works to track and escalate stability issues for both Desktop and Mobile products. Our activities include....

  • Track general stability for all our releases and development channels.
  • Triage Top Crashes to find owners, set priorities, and identify vender outreach issues.
  • Ensure that crash bugs are logged and updated with new information.
  • Identify top tools issues so we can find & fix faster.
  • Identify the first thing we want to do to increase developer awareness and commitment to fixing top crashers.
  • Identify the first thing we want to automate.
  • Working with the Socorro team to get features on the roadmap that help us find and diagnose crash bugs better.
  • Set topcrash threshold criteria.

Top Crashes

There are a number of reports that we triage regularly in order to stay on top of issues, most notably the topcrash reports as linked in the front page of Socorro crash-stats (see top links in the Reports & Tools section of this page). Finding reproducible steps and helping us make them actionable will have a direct and in many cases significant impact on our users due to the volume involved. Not all bugs are fixes in our code. Many are 3rd party issues that can be resolved through outreach and/or blocklisting.

Reports & Tools

Mozilla collects crash data which can be easily viewed using Socorro (a web interface to the crash reports). We use Socorro and its numerous reports to monitor stability rates on a daily basis.

The "Are We Stable Yet?" Stability Dashboard tries to collect all important numbers in one place, with colored indicators for hinting at what they mean.


We also have a list of additional reports that we have developed to help us understand certain crashes and trends. We plan over time to integrate these into Socorro and most of these are on the roadmap already.

Planning

  • CrashKill/Plan
  • We are actively working with the Socorro team on a list of priorities for getting features into Socorro that will help crash analysis. We update the list quarterly.

Work Weeks

Weekly Crashkill Meetings

Up to the end of 2014, weekly meetings were held to talk about stability issues across releases and channels. Those now have been replaced by mailing list communication and ad-hoc meetings before and after releases.

Archived info about meetings

Old Stuff