CrashKill/StabilityWeek2013: Difference between revisions

= Day 4: 08/22/2013 =

We should knock some windows in this airport.

=== Actions ===

= Day 3: 08/21/2013 =

== Socorro: future of dev ==

===Actions===

* redo Vagrant-lite (rhelmer)

* keep dev for now for use as a datastore etc (no real action here)

* build new dev env on the VM dumitru gave us (lonnen)

* Redesign crash reporter to better collect emails addresses.

===Actions===

* [Arun] come back for ideas for design changes in the client

https://etherpad.mozilla.org/ux-stability-workweek (my notes -Arun)

* [bsmedberg] Support classification

* [bsmedberg to follow up] Reduce the junk in the help menu.

* [bsmedberg] Redesign email flow

* Build IDs of final release are highlighted in green on this spreadsheet : https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0Av0LdM1CVycIdDBxVFhqNmNkSnBhVnNSc05vMmhRdXc#gid=4

* bajaj - checklist item for channel to be release-

== B2G Stability Process ==

* communicate the device-specific crash report

* [akeybl] Propose creation of a confidential partner-specific Boot2Gecko::Partner Issues (just like POVB)

== B2G Stability Requirements ==

** KaiRo mentioned to have a test-case where they could crash and pass on the details in tn a bug #  

===Actions===

* [NEEDS BUG, Socorro] report by OS version 1.1.*, 1.1.2, etc.

* [NEEDS BUG, Socorro] surface the buildID to understand geo-specific stability issues

* [NEEDS BUG, bsmedberg] need to check for process responsiveness from an external place (app pings), and also pull system logs when reporting - https://bugzilla.mozilla.org/show_bug.cgi?id=908000

* KaiRo->Fabrice to discuss B2G version number-less reports ({{bug|910836}})

* nhirata/Laura to look at https://bugzilla.mozilla.org/show_bug.cgi?id=895246 (duplicate reports)

** client side bug as well, why are we getting dupes at all?

== Meet with B2G Engineering working on Crashes ==

===Actions===

== High Level CrashKill Review ==

** crash and don't get your tabs back (session restore) [non-actionable?]

* "Soak" time for RCs

== Socorro Brainstorming ==

* Combining signatures associated with the same bug # in Socorro (m:n relationship)

* Could we get some kind of MTBF instead of only crashes per ADI?

* Can we annotate amount of usage of number of pageloads?

* [gps] Annotation on "how many tabs were open?"

* [dmajor] Annotation on OS (maybe also build architecture?) so EMPTY dump reports still get that (and use it in Socorro)

* [ask Cww if to do at all] put "increasing memory swap helps to crash less" on SUMO radar

* [tabled] Investigate with UX/UR of warning users of impending doom and proposing ways out

* Can we determine and eliminate duplicate crash reports (ones that are really just re-submitted)

* We have information on gfx chipset and driver, memory, hardware/cpu but don't use it to narrow down the cause of crashes

** Can potentially compare with information like add-ons that are installed

* [kairo to file] Annotate/show events like Firefox or MS releases on the crash charts? SUMO already has the ability to note "events" on their graphs. They use it for releases as well. Maybe we can steal or learn.

== Automated Stability ==

* Jesse, Rob, :gkw, :tracy, :bc

** Or utilize nightly population by filtering fuzzer test cases(as it sometimes finds security issues), if costs is an issue ?

** We found about ~10bugs a week with running fuzzers on ~100 machines .Fuzzer's run's breakpad,parses to text output, matches to see if its a known bug,checks assertation failures,too much recursion, ref count leaks,inconsistent rendering for a given DOM tree

* QA to look at the output from the bug hunter tool given the current scenario

*[bsmedberg to help with the action needed to resolve the issues] Disconnect between fuzzer bugs and the signature in crash-stat ?  

* gkw, to ensure decoder uploads stacks as well with the bug reports

* Jesse to release part of fuzzzers so nightly population can run it  

* If we can't pass all TBPL tests using ASan/etc, can we enable ASan/etc for the subset of tests that do pass (using test manifests for ASan opt-in)?

* Can we get Nightly ASan builds that can auto-update so adventerous users can use Nightly ASan as their regular browser?

* ('''1''') JSON version of minidump stackwalk

** Goal is to add all sorts of additional data to do identify JIT frame, go back to the caller etc (??)

** Implement in a way that the existing crash-reporter UI should be able to do it when he restarts firefox

** Store a crash-report on user's computer and upload on demand ?

* See above '''bolded''' #s

* [lars] JSON minidump_stackwalk getting deployed

* :nbp,Jesse from the JS team are here

* What are the pain points for JS team related to Socorro ?

* Making flexible Elastic search Queries ? [Adrian]

* [bajaj] Create a bug# to have the total GC crashes on Nightly graphed

* [Laura] to find list of what JS team wants to filter on, determine whether these are common between teams or we need a flexible solution

* Enable search results to be split by custom criteria (e.g. addresses) instead of signature

* [bsmedberg/nbp] Store where we are in JIT code in a page, dump it as part of the crash stack

* [Socorro] exploitable crashes could be marked differently, see above for heuristics

== Radeon Update & Driver Investigation ==

** We need to give their QA to have them engaged? -milan

** Getting to the right person would help

* [bsmedberg] learning kernel debugging for the radeon crash, from our new in-house expert

** having an engineer would make things much much more likely to be resolved, at MS you can even pay $$ for support tickets to be reviewed by egr

* [lsblakk/bsmedberg] highly correlated crashes in FF23.0, discussing in post-mortem

** [milan] to follow up on above bug

* Ran into situations that happen on older cards

** Can we purchase gfx cards before they're EOL'd or when they're released? -milan

* what about issues where millions of users crash once? -bsmedberg

** still impacts overall stability perception, obviously

* [Milan/Anthony/Marc] how to get access to gfx hardware, or access to user computers (remote access?)

** is there a gfx card service like https://appthwack.com/

** [bajaj] bug # incoming

* [bsmedberg] flow chart for developers: how to get the data that you need, next steps in the investigation, etc.

* [Laura/bsmedberg] help get around the legalities involved with crash data access to contributors and external partners (and once those are resolved, the logistics)

** needs subgoals!