CrashKill
General Agenda
Crashkill team works to track and escalate stability issues for both Desktop and Mobile products. Our activities include....
- Track general stability for all our releases and development channels.
- Triage Top Crashes to find owners, set priorities, and identify vender outreach issues.
- Ensure that crash bugs are logged and updated with new information.
- Identify top tools issues so we can find & fix faster.
- Identify the first thing we want to do to increase developer awareness and commitment to fixing top crashers.
- Identify the first thing we want to automate.
- Working with the Socorro team to get features on the roadmap that help us find and diagnose crash bugs better.
Top Crashes
There are a number of bug queries that we triage regularly in order to stay on top of issues. Finding reproducible steps and helping us make them actionable will have a direct and in many cases significant impact on our users due to the volume involved. Not all bugs are fixes in our code. Many are 3rd party issues that can be resolved through outreach and/or blocklisting.
Desktop
- Top crash lists
Mobile
- Top crash lists (Fennec Native only)
Tracking a particular release (Desktop + Mobile)
Reports & Tools
Mozilla collects crash data which can be easily viewed using Socorro (a web interface to the crash reports). We use Socorro and it's numerous reports to monitor stability rates on a daily basis.
- https://crash-stats.mozilla.com/products/Firefox (Desktop)
- https://crash-stats.mozilla.com/products/FennecAndroid (Native)
- https://crash-stats.mozilla.com/products/Fennec (XUL)
We also have a list of additional reports that we have developed to help us understand certain crashes and trends. We plan over time to integrate these into Socorro and most of these are on the roadmap already.
- https://crash-analysis.mozilla.com/chofmann/
- https://crash-analysis.mozilla.com/rkaiser/
- Weekly Crash Bugs Report
- Weekly Socorro Bugs Report
- Explosiveness (in preparation of bug 629062):
- Components (see also bug 697581):
- Desktop: Nightly, Aurora, 10.0 Beta, 9.0 Release, 8.0 Release
- Mobile (Native): Nightly (weekly), Aurora (weekly)
- Mobile (XUL): Nightly (weekly), Aurora (weekly), 10.0 Beta, 9.0 Release, 8.0 Release
- Devices:
- Startup Summary:
- Desktop: Nightly, Aurora, 10.0 Beta, 9.0 Release, 8.0 Release
- Mobile (Native): Nightly, Aurora
- Mobile (XUL): Nightly, Aurora, 10.0 Beta, 9.0 Release, 8.0 Release
- https://wiki.mozilla.org/Temp/
Tools
Planning
Stability Related Projects
Crash Analysis
We have been doing some analysis on fixed crash bugs to better understand how they get fixed. Our results are compiled in this spreadsheet.
10.7 (Lion) crashes
We are keeping track of Flash related hangs and crashes in this wiki page
- Triage
- Historical stuff
Mobile
- Native UI Crashes
- Helpful reports
- Trunk stability
- Weekly reports
- Other stuff we are tracking
- Skiplist bug Bug 683385
Weekly Crashkill Meetings
We meet each week to talk about stability issues across releases and channels.
Logistics
- Monday @ 10:00am PDT
- Dial-in: Audio-only conference# 290
- People with Mozilla phones or softphones please dial x4000 Conf# 290
- US/Toll-free: +1 800 707 2533, (pin 4000) Conf# 290
- US/California/Mountain View: +1 650 903 0800, x4000 Conf# 290
- US/California/San Francisco: +1 415 762 5700, x4000 Conf# 290
- US/Oregon/Portland: +1 971 544 8000, x4000 Conf# 290
- CA/British Columbia/Vancouver: +1 778 785 1540, x4000 Conf# 290
- CA/Ontario/Toronto: +1 416 848 3114, x4000 Conf# 290
- UK/London: +44 (0)207 855 3000, x4000 Conf# 290
- FR/Paris: +33 1 84 88 37 37, x4000 Conf# 290
- Gmail Chat (requires Flash and the Google Talk plugin): paste +1 650 903 0800 into the Gmail Chat box that doesn't look like it accepts phone numbers
- SkypeOut is free if you use the 800 number
- *1 to unmute yourself
- join irc.mozilla.org #crashkill for back channel
- Mountain View conference room "Zombocom"
Meeting notes
- Monday, Feb 13th, 10:00am PST
- Monday, Feb 6th, 10:00am PST
- Monday, Jan 30th, 10:00am PST
- Monday, Jan 23th, 10:00am PST
- Monday, Jan 9th, 10:00am PST
- Monday, Dec 19th, 10:00am PST
- Monday, Dec 12th, 10:00am PST
- Monday, Dec 5th, 10:00am PST
- Monday, Nov 28th, 10:00am PST
- Monday, Nov 21th, 10:00am PST
- Monday, Nov 14th, 10:00am PST
- Monday, Nov 7th, 10:00am PST
- Monday, Oct 3rd, 9:00am PST
- Monday, Sept 26th, 9:00am PST
- Tuesday, Sept 20th, 9:00am PST
- Tuesday, Sept 6th, 10:00am PST
- Monday, Aug 29, 10:00am PST
- Monday, Aug 22, 10:00am PST
- Monday, Aug 15, 10:00am PST
- Monday, Aug 08, 10:00am PST
- Tuesday, Aug 2, 10:00am PST
- Monday, July 18, 10:00am
- Monday, June 27, 10:00am
- Monday, May 2, 10:30am
- Tuesday, August 10, 12:30pm
- Tuesday, May 4, 12:30pm
- Monday, March 1, 3pm PST
- Monday, February 8, 3pm PST
- Monday, February 1, 3pm PST
- Monday, December 14, 3pm PST
- Monday, November 30, 3pm PST
- Monday, November 23, 3pm PST
- Monday, November 16, 3pm PST
- Monday, November 9, 3pm PST
- Monday, November 2, 3pm PST
- Monday, October 26, 3pm PDT
- Monday, October 19, 3pm PDT
- Monday, October 12, 3pm PDT
- Monday, October 5, 3pm PDT
- Monday, September 28, 3pm PDT
Old Stuff
Bug lists
- The CrashKill Bugs (Status Whiteboard: [crashkill])
- 3RD PARTY CrashKill Bugs caused by third parties (Status Whiteboard: [crashkill-thirdparty])
- BLOCKLIST Crashkill Bugs where we're considering blocking the addon/dll (Status Whiteboard: [crashkill-block])
- FIXED CrashKill Bugs where we've provided a fix of some kind, even if just a wallpaper or skidmark (Status Whiteboard: [crashkill-fix])
- DEBUG CrashKill Bugs where we've provide a patch only to help debug the problem (Status Whiteboard: [crashkill-debug])
- List of bugs tracking metrics improvements for CrashKill efforts. (Status Whiteboard: [crashkill-metrics])
- nominate bugs for the 3.6-only DLL blocklist by adding a dependency to bug 525103
- SkipList Bugs and how to make a SkipList request
Analysis
- Firefox 3.5.5 top crashes in the last week
- Firefox 3.6 beta 4 top crashes in the last week
- Correlation Reports
- Crash signatures by module
- chofmann's new crash signatures
- jst's new crash signatures, running daily
- Outreach Bugs tracking our efforts to contact add on vendors who have products causing crashes that we want to kill.
- crash bug trend dashboard some metrics around crash bugs filed, fixed, and resolved through triage.
- How to sort a bug list in topcrash order
- Mac Flash Identifiers - Turns "025105C956638D665850591768FB743D0" into "Flash 10.0.32.18 on PPC"