|
|
| (93 intermediate revisions by 11 users not shown) |
| Line 1: |
Line 1: |
| __NOTOC__
| |
| ==General Agenda== | | ==General Agenda== |
| Crashkill team works to track and escalate stability issues for both Desktop and Mobile products. Our activities include.... | | Crashkill team works to track and escalate stability issues for both Desktop and Mobile products. Our activities include.... |
| Line 10: |
Line 9: |
| * Identify the first thing we want to automate. | | * Identify the first thing we want to automate. |
| * Working with the Socorro team to get features on the roadmap that help us find and diagnose crash bugs better. | | * Working with the Socorro team to get features on the roadmap that help us find and diagnose crash bugs better. |
| | * Set [[CrashKill/Topcrash|topcrash threshold criteria]]. |
|
| |
|
| ==Top Crashes== | | ==Top Crashes== |
|
| |
|
| There are a number of bug queries that we triage regularly in order to stay on top of issues. Finding reproducible steps and helping us make them actionable will have a direct and in many cases significant impact on our users due to the volume involved. Not all bugs are fixes in our code. Many are 3rd party issues that can be resolved through outreach and/or blocklisting. | | There are a number of reports that we triage regularly in order to stay on top of issues, most notably the topcrash reports as linked in the front page of Socorro crash-stats (see top links in the Reports & Tools section of this page). Finding reproducible steps and helping us make them actionable will have a direct and in many cases significant impact on our users due to the volume involved. Not all bugs are fixes in our code. Many are 3rd party issues that can be resolved through outreach and/or blocklisting. |
| | |
| ===Firefox Desktop===
| |
| * Top crash lists
| |
| **[http://bit.ly/vO7SxB Top crashes]
| |
| **[http://bit.ly/xa8sDH Reproducible top crashes]
| |
| **[http://bit.ly/z0wY8v Top hangs]
| |
| **[https://bugzilla.mozilla.org/buglist.cgi?keywords=topcrash&keywords_type=allwords&field0-0-0=status_whiteboard&type0-0-1=notsubstring&field0-0-1=status_whiteboard&resolution=---&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=startupcrash&value0-0-1=%5Bmobile-crash%5D&type0-0-0=notsubstring&value0-0-0=%5Bnative-crash%5D&product=Core&product=Firefox&product=Toolkit&list_id=6869062 Startup top crashes]
| |
| **[http://bit.ly/yrvVAY Reproducible crashes]
| |
| **[https://bugzilla.mozilla.org/buglist.cgi?keywords=crash%2C%20testcase&type0-1-0=notsubstring&keywords_type=allwords&field0-0-0=status_whiteboard&type0-0-0=notsubstring&value0-0-0=%5Bnative-crash%5D&resolution=---&query_format=advanced&product=Core&product=Firefox&product=Toolkit&list_id=6869124 Crashes with testcase]
| |
| **[[CrashKill/TopCrashBlocklist]]
| |
| | |
| * Other notable crash queries
| |
| **[http://bit.ly/GDK8v2 Unconfirmed crashes]
| |
| **[http://bit.ly/GLDEXZ Crash regressions]
| |
| | |
| ===Firefox Metro===
| |
| * Top crash lists
| |
| **[https://bugzilla.mozilla.org/buglist.cgi?keywords=topcrash&keywords_type=allwords&resolution=---&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=%5Bmetro-crash%5D&list_id=6869185 Top crashes]
| |
| **[https://bugzilla.mozilla.org/buglist.cgi?keywords=crash%2C%20reproducible&keywords_type=allwords&resolution=---&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=%5Bmetro-crash%5D&product=Core&product=Firefox%20for%20Metro&list_id=6869285 Reproducible crashes]
| |
| | |
| ===Firefox Mobile===
| |
| * Top crash lists
| |
| **[http://bit.ly/tr4pBc Top crashes]
| |
| **[https://bugzilla.mozilla.org/buglist.cgi?keywords=topcrash%2C%20reproducible&keywords_type=allwords&resolution=---&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=%5Bnative-crash%5D&product=Core&product=Firefox%20for%20Android&list_id=6869268 Reproducible top crashes]
| |
| **[https://bugzilla.mozilla.org/buglist.cgi?keywords=crash%2C%20reproducible&keywords_type=allwords&resolution=---&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=%5Bnative-crash%5D&product=Core&product=Firefox%20for%20Android&list_id=6869223 Reproducible crashes]
| |
| **[https://bugzilla.mozilla.org/buglist.cgi?keywords=crash%2C%20testcase&keywords_type=allwords&resolution=---&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=%5Bnative-crash%5D&product=Core&product=Firefox%20for%20Android&list_id=6869254 Crashes with testcase]
| |
| * Other notable crash queries
| |
| **[https://bugzilla.mozilla.org/buglist.cgi?keywords=crash%2C%20regression&keywords_type=allwords&resolution=---&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=%5Bnative-crash%5D&product=Core&product=Firefox%20for%20Android&list_id=6869306 Crash regressions]
| |
| | |
| ==== Contribute! ====
| |
| * [https://wiki.mozilla.org/CrashKill/Mobile_Process Bug Processing for Mobile]
| |
| | |
| ===Firefox OS===
| |
| * Top crash lists
| |
| **[https://bugzilla.mozilla.org/buglist.cgi?keywords=topcrash&keywords_type=allwords&resolution=---&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=%5Bb2g-crash%5D&list_id=6868997 Top crashes]
| |
| | |
| ===Tracking a particular release===
| |
| * Firefox Desktop & Mobile
| |
| **[https://bugzilla.mozilla.org/buglist.cgi?keywords=crash&keywords_type=allwords&field0-0-0=cf_tracking_firefox22&resolution=---&query_format=advanced&type0-0-0=equals&value0-0-0=%2B&list_id=6869018 Tracking Firefox 22]
| |
| **[https://bugzilla.mozilla.org/buglist.cgi?keywords=crash&keywords_type=allwords&field0-0-0=cf_tracking_firefox23&resolution=---&query_format=advanced&type0-0-0=equals&value0-0-0=%2B&list_id=6869027 Tracking Firefox 23]
| |
| **[https://bugzilla.mozilla.org/buglist.cgi?keywords=crash&keywords_type=allwords&field0-0-0=cf_tracking_firefox24&resolution=---&query_format=advanced&type0-0-0=equals&value0-0-0=%2B&list_id=6869031 Tracking Firefox 24]
| |
| * Firefox OS
| |
| ** [https://bugzilla.mozilla.org/buglist.cgi?keywords=crash&keywords_type=allwords&status_whiteboard_type=allwordssubstr&resolution=---&query_format=advanced&f1=cf_blocking_b2g&o1=equals&v1=leo%2B&f2=cf_status_b2g18&o2=notequals&v2=fixed&f3=cf_status_b2g18&o3=notequals&v3=verified&o4=notequals&f4=cf_status_b2g18&v4=unaffected&product=Boot2Gecko&product=Core&list_id=6869468 Tracking 1.1 leo]
| |
| ** [https://bugzilla.mozilla.org/buglist.cgi?keywords=crash&keywords_type=allwords&status_whiteboard_type=allwordssubstr&resolution=---&query_format=advanced&f1=cf_blocking_b2g&o1=equals&v1=hd%2B&f2=cf_status_b2g_1_1_hd&o2=notequals&v2=fixed&f3=cf_status_b2g_1_1_hd&o3=notequals&v3=verified&o4=notequals&f4=cf_status_b2g_1_1_hd&v4=unaffected&product=Boot2Gecko&product=Core&list_id=6869498 Tracking 1.1 HD]
| |
|
| |
|
| == Reports & Tools == | | == Reports & Tools == |
| Line 63: |
Line 19: |
| Mozilla collects crash data which can be easily viewed using Socorro (a web interface to the crash reports). We use Socorro and its numerous reports to monitor stability rates on a daily basis. | | Mozilla collects crash data which can be easily viewed using Socorro (a web interface to the crash reports). We use Socorro and its numerous reports to monitor stability rates on a daily basis. |
| * https://crash-stats.mozilla.com/products/Firefox (Firefox Desktop) | | * https://crash-stats.mozilla.com/products/Firefox (Firefox Desktop) |
| * https://crash-stats.mozilla.com/products/MetroFirefox (Firefox Metro)
| | * https://crash-stats.mozilla.com/products/FennecAndroid (Firefox for Android) |
| * https://crash-stats.mozilla.com/products/FennecAndroid (Firefox Mobile) | |
| * https://crash-stats.mozilla.com/products/B2G (Firefox OS) | | * https://crash-stats.mozilla.com/products/B2G (Firefox OS) |
| | |
| | The [http://arewestableyet.com/ "Are We Stable Yet?" Stability Dashboard] tries to collect all important numbers in one place, with colored indicators for hinting at what they mean. |
|
| |
|
|
| |
|
| Line 72: |
Line 29: |
| * https://crash-analysis.mozilla.com/chofmann/ | | * https://crash-analysis.mozilla.com/chofmann/ |
| * https://crash-analysis.mozilla.com/rkaiser/ (see [https://crash-analysis.mozilla.com/rkaiser/0000.overview.html Overview] for always-current links) | | * https://crash-analysis.mozilla.com/rkaiser/ (see [https://crash-analysis.mozilla.com/rkaiser/0000.overview.html Overview] for always-current links) |
| ** [https://crash-analysis.mozilla.com/rkaiser/crash-bugs/2013.weeklybugs.html Weekly Crash Bugs Report] | | ** Explosiveness, Weekly bug stats, FxOS crashes by B2G version, Flash topcrashes by version, and other specialty reports |
| ** [https://crash-analysis.mozilla.com/rkaiser/socorro-bugs/2013.weeklybugs.html Weekly Socorro Bugs Report]
| |
| ** '''Explosiveness''' (in preparation of {{bug|629062}}):
| |
| *** Desktop: [https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.firefox.24.explosiveness.html 24.0*], [https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.firefox.23.explosiveness.html 23.0*], [https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.firefox.22.explosiveness.html 22.0*], [https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.firefox.21.explosiveness.html 21.0*], [https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.firefox.17.explosiveness.html 17.0*]
| |
| *** Mobile: [https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.fennecandroid.nightly.explosiveness.html Nightly], [https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.fennecandroid.aurora.explosiveness.html Aurora], [https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.fennecandroid.22.explosiveness.html 22.0*], [https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.fennecandroid.21.explosiveness.html 21.0*]
| |
| ** '''Components''' (see also {{bug|697581}}):
| |
| *** Desktop: [https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.firefox.nightly.components.html Nightly], [https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.firefox.aurora.components.html Aurora], [https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.firefox.beta.22.0.components.html 22.0 Beta], [https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.firefox.release.21.0.components.html 21.0 Release], [https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.firefox.esr.17.0.components.html 17.0 ESR]
| |
| *** Mobile: [https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.fennecandroid.nightly.components.html Nightly] ([https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.fennecandroid.nightly.components.weekly.html weekly]), [https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.fennecandroid.aurora.components.html Aurora] ([https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.fennecandroid.aurora.components.weekly.html weekly]), [https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.fennecandroid.beta.22.0.components.html 22.0 Beta], [https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.fennecandroid.release.21.0.components.html 21.0 Release]
| |
| ** '''Devices''' (see also {{bug|687115}}):
| |
| *** Mobile: [https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.fennecandroid.nightly.devices.html Nightly] ([https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.fennecandroid.nightly.devices.weekly.html weekly]), [https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.fennecandroid.aurora.devices.html Aurora] ([https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.fennecandroid.aurora.devices.weekly.html weekly]), [https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.fennecandroid.beta.22.0.devices.html 22.0 Beta], [https://crash-analysis.mozilla.com/rkaiser/{{#time: Y-m-d | -1 day}}/{{#time: Y-m-d | -1 day}}.fennec.release.21.0.devices.html 21.0 Release]
| |
| ** '''Startup Summary''':
| |
| *** Desktop: [https://crash-analysis.mozilla.com/rkaiser/firefox.nightly.startupsummary.html Nightly], [https://crash-analysis.mozilla.com/rkaiser/firefox.aurora.startupsummary.html Aurora], [https://crash-analysis.mozilla.com/rkaiser/firefox.beta.22.0.startupsummary.html 22.0 Beta], [https://crash-analysis.mozilla.com/rkaiser/firefox.release.21.0.startupsummary.html 21.0 Release], [https://crash-analysis.mozilla.com/rkaiser/firefox.esr.17.0.startupsummary.html 17.0 ESR]
| |
| *** Mobile: [https://crash-analysis.mozilla.com/rkaiser/fennecandroid.nightly.startupsummary.html Nightly], [https://crash-analysis.mozilla.com/rkaiser/fennecandroid.aurora.startupsummary.html Aurora], [https://crash-analysis.mozilla.com/rkaiser/fennecandroid.beta.22.0.startupsummary.html 22.0 Beta], [https://crash-analysis.mozilla.com/rkaiser/fennecandroid.release.21.0.startupsummary.html 21.0 Release]
| |
| * https://crash-analysis.mozilla.com/bsmedberg/ | | * https://crash-analysis.mozilla.com/bsmedberg/ |
| ** https://crash-analysis.mozilla.com/bsmedberg/flash-summary.html (summary breakdown of crashes and hangs by Flash version) | | ** https://crash-analysis.mozilla.com/bsmedberg/flash-summary.html (summary breakdown of crashes and hangs by Flash version) |
| ** [[CrashKill/Flash Breakdown Data]] | | ** [[CrashKill/Flash Breakdown Data]] |
| | * Reporting Malware (external link): https://www.techsupportalert.com/content/how-report-malware-or-false-positives-multiple-antivirus-vendors.htm |
|
| |
|
| == Planning == | | == Planning == |
| Line 93: |
Line 39: |
| * We are actively working with the Socorro team on a list of [https://wiki.mozilla.org/CrashKill/Plan/Priorities priorities] for getting features into Socorro that will help crash analysis. We update the list quarterly. | | * We are actively working with the Socorro team on a list of [https://wiki.mozilla.org/CrashKill/Plan/Priorities priorities] for getting features into Socorro that will help crash analysis. We update the list quarterly. |
|
| |
|
| == Stability Related Projects == | | ==Work Weeks== |
| ===Crash Analysis===
| | *[[CrashKill/WorkWeek2012|Stability Work Week 2012]] (Notes) |
| We have been doing some analysis on fixed crash bugs to better understand how they get fixed. Our results are compiled in this spreadsheet.
| | *[[CrashKill/StabilityWeek2013|Stability Week 2013]] (Notes) |
| * [https://docs.google.com/spreadsheet/ccc?key=0AqfkEC1pmwTidFRmUjZkYVhLV3gyWDJRTFhIZUpwZmc&hl=en_US#gid=0 Spreadsheet tracking crash bug stats] | |
| ===10.7 (Lion) crashes===
| |
| * [http://bit.ly/ny2xan Open 10.7 crashes] | |
| * [http://bit.ly/oqa7WN Fixed 10.7 crashes]
| |
| ===Triage===
| |
| * [https://wiki.mozilla.org/CrashKill/DesktopTriage Desktop Triage]
| |
|
| |
|
| ==Weekly Crashkill Meetings== | | ==Weekly Crashkill Meetings== |
| We meet each week to talk about stability issues across releases and channels.
| | Up to the end of 2014, weekly meetings were held to talk about stability issues across releases and channels. Those now have been replaced by mailing list communication and ad-hoc meetings before and after releases. |
| ===Logistics===
| |
| * Monday @ 10:00am PT
| |
| * Mozilla Mountain View: Zombocom Conference Room, 3rd floor
| |
| * Vidyo: Stability
| |
| * Meeting notes & queries: https://wiki.mozilla.org/CrashKill
| |
| * join irc.mozilla.org #crashkill for back channel
| |
| <createbox>
| |
| align=left
| |
| type=create
| |
| preload=CrashKill/CrashKill-Template
| |
| default={{#time: Y-m-d | monday}}
| |
| prefix=CrashKill/
| |
| </createbox>
| |
| | |
| ===Meeting notes===
| |
| ====2013====
| |
| * [[CrashKill/2013-06-17|Monday, June 17th, 10:00am PDT]]
| |
| * [[CrashKill/2013-06-10|Monday, June 10th, 10:00am PDT]]
| |
| * [[CrashKill/2013-06-03|Monday, June 3rd, 10:00am PDT]]
| |
| * Monday, May 27th, 10:00am PDT - no meeting due to US holiday
| |
| * [[CrashKill/2013-05-20|Monday, May 20th, 10:00am PDT]]
| |
| * [[CrashKill/2013-05-13|Monday, May 13th, 10:00am PDT]]
| |
| * [[CrashKill/2013-05-06|Monday, May 6th, 10:00am PDT]]
| |
| * [[CrashKill/2013-04-29|Monday, April 29nd, 10:00am PDT]]
| |
| * [[CrashKill/2013-04-22|Monday, April 22nd, 10:00am PDT]]
| |
| * [[CrashKill/2013-04-15|Monday, April 15th, 10:00am PDT]]
| |
| * [[CrashKill/2013-04-08|Monday, April 8th, 10:00am PDT]]
| |
| * [[CrashKill/2013-04-01|Monday, April 1st, 10:00am PDT]]
| |
| * [[CrashKill/2013-03-25|Monday, March 25th, 10:00am PDT]]
| |
| * [[CrashKill/2013-03-18|Monday, March 18th, 10:00am PDT]]
| |
| * [[CrashKill/2013-03-11|Monday, March 11th, 10:00am PDT]]
| |
| * [[CrashKill/2013-03-04|Monday, March 4th, 10:00am PST]]
| |
| * [[CrashKill/2013-02-25|Monday, February 25th, 10:00am PST]]
| |
| * [[CrashKill/2013-02-18|Monday, February 18th, 10:00am PST: Meeting Cancelled]]
| |
| * [[CrashKill/2013-02-11|Monday, February 11th, 10:00am PST]]
| |
| * [[CrashKill/2013-02-04|Monday, February 4th, 10:00am PST]]
| |
| * [[CrashKill/2013-01-28|Monday, January 28th, 10:00am PST]]
| |
| * [[CrashKill/2013-01-14|Monday, January 14th, 10:00am PST]]
| |
| * [[CrashKill/2013-01-07|Monday, January 7th, 10:00am PST]]
| |
| | |
| ====2012====
| |
| * [[CrashKill/2012-12-31|Monday, December 31st, 10:00am PST]]
| |
| * [[CrashKill/2012-12-17|Monday, December 17th, 10:00am PST]]
| |
| * [[CrashKill/2012-12-10|Monday, December 10th, 10:00am PST]]
| |
| * [[CrashKill/2012-12-03|Monday, December 3rd, 10:00am PST]]
| |
| * [[CrashKill/2012-11-26|Monday, November 26th, 10:00am PST]]
| |
| * [[CrashKill/2012-11-19|Monday, November 19th, 10:00am PST]]
| |
| * [[CrashKill/2012-11-12|Monday, November 12th, 10:00am PST]]
| |
| * [[CrashKill/2012-11-05|Monday, November 5th, 10:00am PST]]
| |
| * [[CrashKill/2012-10-29|Monday, October 29th, 10:00am PST]]
| |
| * [[CrashKill/2012-10-22|Monday, October 22nd, 10:00am PST]]
| |
| * [[CrashKill/2012-10-15|Monday, October 15th, 10:00am PST]]
| |
| * [[CrashKill/2012-10-08|Monday, October 8th, 10:00am PST]]
| |
| * [[CrashKill/2012-10-01|Monday, October 1st, 10:00am PST]]
| |
| * [[CrashKill/2012-09-24|Monday, September 24th, 10:00am PST]]
| |
| * [[CrashKill/2012-09-17|Monday, September 17th, 10:00am PST]]
| |
| * [[CrashKill/2012-09-03|Monday, September 3rd, 10:00am PST]]
| |
| * [[CrashKill/2012-08-27|Monday, August 27th, 10:00am PST]]
| |
| * [[CrashKill/2012-08-20|Monday, August 20th, 10:00am PST]]
| |
| * [[CrashKill/2012-08-13|Monday, August 13th, 10:00am PST]]
| |
| * [[CrashKill/2012-08-06|Monday, August 6th, 10:00am PST]]
| |
| * [[CrashKill/2012-07-23|Monday, July 23rd, 10:00am PST]]
| |
| * [[CrashKill/2012-07-16|Monday, July 16th, 10:00am PST]]
| |
| * [[CrashKill/2012-07-09|Monday, July 9th, 10:00am PST]]
| |
| * [[CrashKill/2012-07-02|Monday, July 2st, 10:00am PST]]
| |
| * [[CrashKill/2012-06-04|Monday, June 4th, 10:00am PST]]
| |
| * [[CrashKill/WorkWeek2012|Stability Work Week - Jun 11-15, 2012]]
| |
| * [[CrashKill/2012-06-04|Monday, June 4th, 10:00am PST]]
| |
| * [[CrashKill/2012-05-21|Monday, May 21th, 10:00am PST]]
| |
| * [[CrashKill/2012-05-14|Monday, May 14th, 10:00am PST]]
| |
| * [[CrashKill/2012-05-07|Monday, May 7th, 10:00am PST]]
| |
| * [[CrashKill/2012-04-30|Monday, April 30, 10:00am PST]]
| |
| * [[CrashKill/2012-04-16|Monday, April 16, 10:00am PST]]
| |
| * [[CrashKill/2012-04-09|Monday, April 9, 10:00am PST]]
| |
| * [[CrashKill/2012-04-02|Monday, April 2, 10:00am PST]]
| |
| * [[CrashKill/2012-03-26|Monday, Mar 26th, 10:00am PST]]
| |
| * [[CrashKill/2012-03-19|Monday, Mar 19th, 10:00am PST]]
| |
| * [[CrashKill/2012-03-12|Monday, Mar 12th, 10:00am PST]]
| |
| * [[CrashKill/2012-02-27|Monday, Feb 27th, 10:00am PST]]
| |
| * [[CrashKill/2012-02-13|Monday, Feb 13th, 10:00am PST]]
| |
| * [[CrashKill/2012-02-06|Monday, Feb 6th, 10:00am PST]]
| |
| * [[CrashKill/2012-01-30|Monday, Jan 30th, 10:00am PST]]
| |
|
| |
|
| ===Newsletter===
| | [[CrashKill/Meetings|Archived info about meetings]] |
| * [[CrashKill/2012-09-04|Tuesday September 4th, 2012]]
| |
|
| |
|
| ==Old Stuff== | | ==Old Stuff== |
| * [[CrashKill/Historical| Old links, queries and info]] | | * [[CrashKill/Historical| Old links, queries and info]] |