CrashKill
General Agenda
Crashkill team works to track and escalate stability issues for both Desktop and Mobile products. Our activities include....
- Track general stability for all our releases and development channels.
- Triage Top Crashes to find owners, set priorities, and identify vender outreach issues.
- Ensure that crash bugs are logged and updated with new information.
- Identify top tools issues so we can find & fix faster.
- Identify the first thing we want to do to increase developer awareness and commitment to fixing top crashers.
- Identify the first thing we want to automate.
- Working with the Socorro team to get features on the roadmap that help us find and diagnose crash bugs better.
Top Crashes
There are a number of bug queries that we triage regularly in order to stay on top of issues. Finding reproducible steps and helping us make them actionable will have a direct and in many cases significant impact on our users due to the volume involved. Not all bugs are fixes in our code. Many are 3rd party issues that can be resolved through outreach and/or blocklisting.
Firefox Desktop
- Top crash lists
- Other notable crash queries
Firefox Metro
- Top crash lists
Firefox Mobile
- Top crash lists
- Other notable crash queries
Contribute!
Firefox OS
- Top crash lists
Tracking a particular release
- Firefox Desktop & Mobile
- Firefox OS
Reports & Tools
Mozilla collects crash data which can be easily viewed using Socorro (a web interface to the crash reports). We use Socorro and its numerous reports to monitor stability rates on a daily basis.
- https://crash-stats.mozilla.com/products/Firefox (Firefox Desktop)
- https://crash-stats.mozilla.com/products/MetroFirefox (Firefox Metro)
- https://crash-stats.mozilla.com/products/FennecAndroid (Firefox Mobile)
- https://crash-stats.mozilla.com/products/B2G (Firefox OS)
We also have a list of additional reports that we have developed to help us understand certain crashes and trends. We plan over time to integrate these into Socorro and most of these are on the roadmap already.
- https://crash-analysis.mozilla.com/chofmann/
- https://crash-analysis.mozilla.com/rkaiser/ (see Overview for always-current links)
- Weekly Crash Bugs Report
- Weekly Socorro Bugs Report
- Explosiveness (in preparation of bug 629062):
- Components (see also bug 697581):
- Devices (see also bug 687115):
- Startup Summary:
- Desktop: Nightly, Aurora, 22.0 Beta, 21.0 Release, 17.0 ESR
- Mobile: Nightly, Aurora, 22.0 Beta, 21.0 Release
- https://crash-analysis.mozilla.com/bsmedberg/
- https://crash-analysis.mozilla.com/bsmedberg/flash-summary.html (summary breakdown of crashes and hangs by Flash version)
- CrashKill/Flash Breakdown Data
Planning
- CrashKill/Plan
- We are actively working with the Socorro team on a list of priorities for getting features into Socorro that will help crash analysis. We update the list quarterly.
Stability Related Projects
Crash Analysis
We have been doing some analysis on fixed crash bugs to better understand how they get fixed. Our results are compiled in this spreadsheet.
10.7 (Lion) crashes
Triage
Weekly Crashkill Meetings
We meet each week to talk about stability issues across releases and channels.
Logistics
- Monday @ 10:00am PT
- Mozilla Mountain View: Zombocom Conference Room, 3rd floor
- Vidyo: Stability
- Meeting notes & queries: https://wiki.mozilla.org/CrashKill
- join irc.mozilla.org #crashkill for back channel
<createbox> align=left type=create preload=CrashKill/CrashKill-Template default=2025-07-28 prefix=CrashKill/ </createbox>
Meeting notes
2013
- Monday, June 17th, 10:00am PDT
- Monday, June 10th, 10:00am PDT
- Monday, June 3rd, 10:00am PDT
- Monday, May 27th, 10:00am PDT - no meeting due to US holiday
- Monday, May 20th, 10:00am PDT
- Monday, May 13th, 10:00am PDT
- Monday, May 6th, 10:00am PDT
- Monday, April 29nd, 10:00am PDT
- Monday, April 22nd, 10:00am PDT
- Monday, April 15th, 10:00am PDT
- Monday, April 8th, 10:00am PDT
- Monday, April 1st, 10:00am PDT
- Monday, March 25th, 10:00am PDT
- Monday, March 18th, 10:00am PDT
- Monday, March 11th, 10:00am PDT
- Monday, March 4th, 10:00am PST
- Monday, February 25th, 10:00am PST
- Monday, February 18th, 10:00am PST: Meeting Cancelled
- Monday, February 11th, 10:00am PST
- Monday, February 4th, 10:00am PST
- Monday, January 28th, 10:00am PST
- Monday, January 14th, 10:00am PST
- Monday, January 7th, 10:00am PST
2012
- Monday, December 31st, 10:00am PST
- Monday, December 17th, 10:00am PST
- Monday, December 10th, 10:00am PST
- Monday, December 3rd, 10:00am PST
- Monday, November 26th, 10:00am PST
- Monday, November 19th, 10:00am PST
- Monday, November 12th, 10:00am PST
- Monday, November 5th, 10:00am PST
- Monday, October 29th, 10:00am PST
- Monday, October 22nd, 10:00am PST
- Monday, October 15th, 10:00am PST
- Monday, October 8th, 10:00am PST
- Monday, October 1st, 10:00am PST
- Monday, September 24th, 10:00am PST
- Monday, September 17th, 10:00am PST
- Monday, September 3rd, 10:00am PST
- Monday, August 27th, 10:00am PST
- Monday, August 20th, 10:00am PST
- Monday, August 13th, 10:00am PST
- Monday, August 6th, 10:00am PST
- Monday, July 23rd, 10:00am PST
- Monday, July 16th, 10:00am PST
- Monday, July 9th, 10:00am PST
- Monday, July 2st, 10:00am PST
- Monday, June 4th, 10:00am PST
- Stability Work Week - Jun 11-15, 2012
- Monday, June 4th, 10:00am PST
- Monday, May 21th, 10:00am PST
- Monday, May 14th, 10:00am PST
- Monday, May 7th, 10:00am PST
- Monday, April 30, 10:00am PST
- Monday, April 16, 10:00am PST
- Monday, April 9, 10:00am PST
- Monday, April 2, 10:00am PST
- Monday, Mar 26th, 10:00am PST
- Monday, Mar 19th, 10:00am PST
- Monday, Mar 12th, 10:00am PST
- Monday, Feb 27th, 10:00am PST
- Monday, Feb 13th, 10:00am PST
- Monday, Feb 6th, 10:00am PST
- Monday, Jan 30th, 10:00am PST