CrashKill

General Agenda

Crashkill team works to track and escalate stability issues for both Desktop and Mobile products. Our activities include....

• Track general stability for all our releases and development channels.
• Triage Top Crashes to find owners, set priorities, and identify vender outreach issues.
• Ensure that crash bugs are logged and updated with new information.
• Identify top tools issues so we can find & fix faster.
• Identify the first thing we want to do to increase developer awareness and commitment to fixing top crashers.
• Identify the first thing we want to automate.
• Working with the Socorro team to get features on the roadmap that help us find and diagnose crash bugs better.

Top Crashes

There are a number of bug queries that we triage regularly in order to stay on top of issues. Finding reproducible steps and helping us make them actionable will have a direct and in many cases significant impact on our users due to the volume involved. Not all bugs are fixes in our code. Many are 3rd party issues that can be resolved through outreach and/or blocklisting.

Desktop

• Top crash lists
  • Top crashes
  • Reproducible top crashes
  • Top hangs
  • Startup top crashes
  • Reproducible crashes
  • CrashKill/TopCrashBlocklist

• Other notable crash queries
  • Unconfirmed crashes
  • Crash regressions

Mobile

• Top crash lists (Fennec Native only)
  • Top crashes

Contribute!

• Bug Processing for Mobile

Tracking a particular release (Desktop + Mobile)

• Tracking Firefox 14
• Tracking Firefox 15
• Tracking Firefox 16

Reports & Tools

Mozilla collects crash data which can be easily viewed using Socorro (a web interface to the crash reports). We use Socorro and it's numerous reports to monitor stability rates on a daily basis.

• https://crash-stats.mozilla.com/products/Firefox (Desktop)
• https://crash-stats.mozilla.com/products/FennecAndroid (Native)
• https://crash-stats.mozilla.com/products/Fennec (XUL)

We also have a list of additional reports that we have developed to help us understand certain crashes and trends. We plan over time to integrate these into Socorro and most of these are on the roadmap already.

• https://crash-analysis.mozilla.com/chofmann/
• https://crash-analysis.mozilla.com/rkaiser/ (see Overview for always-current links)
  • Weekly Crash Bugs Report
  • Weekly Socorro Bugs Report
  • Explosiveness (in preparation of bug 629062):
    • Desktop: 17.0*, 16.0*, 15.0*, 14.0*, 13.0*
    • Mobile (Native): Nightly, Aurora, 15.0*, 14.0*
    • Mobile (XUL): Nightly, Aurora, 14.0*, 10.0*
  • Components (see also bug 697581):
    • Desktop: Nightly, Aurora, 15.0 Beta, 14.0 Release, 13.0 Release
    • Mobile (Native): Nightly (weekly), Aurora (weekly), 15.0 Beta, 14.0 Release
    • Mobile (XUL): Nightly (weekly), Aurora (weekly), 14.0 Beta, 10.0 ESR
  • Devices (see also bug 687115):
    • Mobile (Native): Nightly (weekly), Aurora (weekly), 15.0 Beta (weekly), 14.0 Release
    • Mobile (XUL): Nightly (weekly), Aurora (weekly), 14.0 Beta (weekly), 10.0 ESR
  • Startup Summary:
    • Desktop: Nightly, Aurora, 15.0 Beta, 14.0 Release, 13.0 Release
    • Mobile (Native): Nightly, Aurora, 15.0 Beta, 14.0 Release
    • Mobile (XUL): Nightly, Aurora, 14.0 Beta, 10.0 ESR
• https://crash-analysis.mozilla.com/bsmedberg/
  • https://crash-analysis.mozilla.com/bsmedberg/flash-summary.html (summary breakdown of crashes and hangs by Flash version)
  • CrashKill/Flash Breakdown Data

Planning

• CrashKill/Plan
• We are actively working with the Socorro team on a list of priorities for getting features into Socorro that will help crash analysis. We update the list quarterly.

Stability Related Projects

Crash Analysis

We have been doing some analysis on fixed crash bugs to better understand how they get fixed. Our results are compiled in this spreadsheet.

• Spreadsheet tracking crash bug stats

10.7 (Lion) crashes

• Open 10.7 crashes
• Fixed 10.7 crashes

Triage

• Desktop Triage

Weekly Crashkill Meetings

We meet each week to talk about stability issues across releases and channels.

Logistics

• Monday @ 10:00am PT
• Mozilla Mountain View: Zombocom Conference Room, 3rd floor
• Vidyo: Stability
• Meeting notes & queries: https://wiki.mozilla.org/CrashKill
• join irc.mozilla.org #crashkill for back channel

Meeting notes

• Monday, October 1st, 10:00am PST
• Monday, September 24th, 10:00am PST
• Monday, September 17th, 10:00am PST
• Monday, September 3rd, 10:00am PST
• Monday, August 27th, 10:00am PST
• Monday, August 20th, 10:00am PST
• Monday, August 13th, 10:00am PST
• Monday, August 6th, 10:00am PST
• Monday, July 23rd, 10:00am PST
• Monday, July 16th, 10:00am PST
• Monday, July 9th, 10:00am PST
• Monday, July 2st, 10:00am PST
• Monday, June 4th, 10:00am PST
• Stability Work Week - Jun 11-15, 2012
• Monday, June 4th, 10:00am PST
• Monday, May 21th, 10:00am PST
• Monday, May 14th, 10:00am PST
• Monday, May 7th, 10:00am PST
• Monday, April 30, 10:00am PST
• Monday, April 16, 10:00am PST
• Monday, April 9, 10:00am PST
• Monday, April 2, 10:00am PST
• Monday, Mar 26th, 10:00am PST
• Monday, Mar 19th, 10:00am PST
• Monday, Mar 12th, 10:00am PST
• Monday, Feb 27th, 10:00am PST
• Monday, Feb 13th, 10:00am PST
• Monday, Feb 6th, 10:00am PST
• Monday, Jan 30th, 10:00am PST

Newsletter

• Tuesday September 4th, 2012

Old Stuff

• Old links, queries and info