CrashKill/Correlations

about correlations and their importance in looking at crash data

_TOC

Intro

the basic set of data in every report is

  crash signature
  url  --- due to privacy concerns we treat the specific site info about the place the user was visting as confidential
  uuid_url  -- link to the crash report  details
  client_crash_date  --    date_processed
       correlation of date to sets of crashes is interesting to us for events like
       https://bug519039.bugzilla.mozilla.org/attachment.cgi?id=403459
       and https://bugzilla.mozilla.org/show_bug.cgi?id=519039
  last_crash
       helps us to understand a bit about relability for each user
  product
  version
  build
  branch
        correlation of these helps us to understand which releases have the problem,
        and which don't given enough testing use
  os_name
  os_version
  cpu_name
          ditto for correlating to which kinds of systems and os'es might come into play
  address
          crashing address helps to correlate some of the signatures and stack traces
  bug_list
           ties of crash signatures to bugs where we track the problem
  user_comment
           all kind of user feedback we get when they submit crashes

then we have the module list of what was loaded into memory

and the stack trace. (the stack traces could be analyzed like signals to to better correlation of crashy code)

Modules, Addon, and Plugin Correlations

dbaron's amazing work has produced this

http://people.mozilla.com/crash_analysis/

to use it

pick a date

http://people.mozilla.com/crash_analysis/20091119/

then open up one of the files depending on which kind of corelation you are looking for and which version of firefox you are looking at.

for bug https://bugzilla.mozilla.org/show_bug.cgi?id=42771 we don't know what to suspect so we might have to look at several files, but the thing we will use to search is the stack signature

open the 20091119_Firefox-interesting-addons.txt file and search for js_GetGCThingTraceKind

to see:

 js_GetGCThingTraceKind|EXC_ARITHMETIC / EXC_I386_DIV (13 crashes)
     8% (1/13) vs.   0% (9/17895) {336dc353-5272-420c-84e7-ba1f3c9c2aeb}
     8% (1/13) vs.   0% (36/17895) lazarus@interclue.com (Lazarus: Form Recovery, https://addons.mozilla.org/addon/6984)

...

then see that there are no good addon correlations.

then do the same for plugins and modules and number of cores.