View source for Socorro:PRD 2.x

{{DRAFT}}

= Overview =
In Socorro 1.7 we move to Hbase for storage, and in 1.8 we move the processors too and turn off throttling for good.  

Socorro 2.x will allow us to take advantage of the capabilities of Hbase and pass those on to the end users.

= Process = 
* Interview users, gather requirements
* Synthesize user requirements, prepare draft PRD 
* Develop mocks/wireframes (week of June 7)
* Get input on PRD from the broader community (begin week of June 14) '''(we are here)'''
** Summit talk
** Blog
* Finalize PRD with community input (August)
* Set dates and milestones for these features in the [https://wiki.mozilla.org/SocorroRoadmap2010 Socorro Roadmap] (end August)

= Requirements Gathering =
We interviewed Socorro users from the Firefox, Platform and QA teams, as well as Mozilla Messaging (for Thunderbird) about their current use of the product.
* [https://wiki.mozilla.org/Socorro:Planning_2.0 Interview prompt sheet]
* [https://wiki.mozilla.org/Socorro:PRD_Interviews Interview notes summaries]

= Required Features = 

== New / explosive / critical crash tracking == 
Users of Socorro want to be able to answer the following questions:
* What common new crashes are there?  
* What crashes have started occurring with increased frequency?
* What date / buildid did a crash first occur?
* When (date / buildid) did a crash spike in frequency?
* What new crashes have appeared that do not have a bugzilla bug associated with them?

Definitions:
* Explosive crashes are ones that have experienced a sharp upward change in frequency.  Exact formula needs to be developed, combining increase in volume and rank. 
* New crashes have no associated bugzilla id.
* Startup crashes occur within one minute of startup.

Requirements:
* For each crash, list date of first occurrence.
* Allow sorting of crashes by date they first occurred.
* If a sharp increase in volume occurs, record the date and buildid
* When explosive crashes are detected, a notification should be emailed to mozilla.dev.tree-management. 
* Allow sorting by / reporting of signatures that do not have an associated bugzilla bug.
* In crash lists (topcrashers/search results), highlight new, explosive, and startup crashes.

=== Mocks ===
[[File:Explosive_signatures.png|400px]]

== Better search ==
=== Full text search ===
Requirements:
* Enable  full text indexing of any text in a crash or crash metadata. Users should then be able to search against any of this text without knowing which field it is in.  
* Perform partial matches.  
* Rank results by relevance.

Implementation:
* This will require a FTS product in front of HBase, likely Lucene.

=== Template search === 
Requirements:
* Allow search using a template that allows exact matching against every field in a crash.  (This should be similar to the way bugzilla advanced search works.)

==== Mocks ====
[[File:Search_fields.png|400px]]
[[File:Search_query.png|400px]]

== Faceted search ==
Faceted search allows the categorization of a set of crashes along a pre-defined set of dimensions.  Given a working dataset, which might be, for example, topcrashers for a particular build, a set of search results, or the set of crashes for a particular signature, users should be able to drill down through these crashes along various dimensions.

=== Faceted signature search ===
Given a set of signatures (top crashers or search results), see them categorized and navigate by these categories.  
* Product version
* Buildid
* OS 
* Component in which the crash occurred

=== Faceted crash search ===
Crashes that are gathered under a particular signature have already effectively been categorized under that signature.  Other aspects of the crashes may differ, for example, the rest of the stack trace, operating system, loaded plugins, and extensions.

Requirements:
* Given a set of crashes for a signature, see them categorized and navigate by these categories. Categories may include:
** Product version
** buildid
** OS
** Time since startup (<1, <5m, >5m)
** Loaded plugins
*** Plugins by version
** Extensions
** Stack frames (how to represent this?)
** Address of crash
** Component in which the crash occurred
** DLL
** Caller (possible to show call tree?)
* Categories should show numbers in each category, and possible values should be listed in descending order of magnitude.
* Generate standard reports of crashes by correlation in all of these categories

===Mocks===
[[File:Signature_filter_closed.png|200px|Faceted signature search - facet closed]]
[[File:Signature_filter_open.png|200px|Faceted signature search - facet open]] 
[[File:Report_filter.png|200px|Faceted crash search]]

== MapReduce UI ==
Requirements:
* Provide UI access to allow (some?) users to write and schedule their own MapReduce jobs.
* Provide a mechanism for saving these jobs and scheduling them regularly
* Provide a way for users to share the results of these jobs
* Provide user training and documentation on how to write MapReduce jobs in JavaScript.

=== Mocks ===
[[File:Mapreduce_all.png|200px|Show all MR jobs]]
[[File:Mapreduce_one_job.png|200px|Show one MR job]]
[[File:Mapreduce_new_job.png|200px|Create new MR job]]

=== Community feedback ===
Cloudera suggests we use HUE to implement this feature:
* [http://www.cloudera.com/blog/2010/07/whats-new-in-cdh3b2-hue What is HUE]
* [http://www.cloudera.com/blog/2010/07/developing-applications-for-hue Developing Apps for HUE]
* [https://issues.cloudera.org/browse/HUE Issue tracker]

== User post-crash experience improvements ==
This feature is growing out of {{bug|411425}} to be a project involving changes to about:crashes, SUMO, and Socorro.  See [https://wiki.mozilla.org/PostCrash PostCrash User Engagement].

== Other features ==

=== Filtering ===
* Filter topcrashers view by hang/crash
* Filter topcrashers by product/thirdparty
* Filter topcrashers to only show those without associated bugs
* Filter crashes for a signature by those with email addresses supplied (covered by faceted search)
* Filter crashes for a signature by those with comments (covered by faceted search)

=== Plugins, extensions, DLLs ===
* Supply human readable names for plugins 
* Supply human readable names for addons
* Build a database of known DLLs and identify them where known (e.g. Windows 7, Windows XP "official" DLLs)

=== Signature page improvements ===
(This is for http://crash-stats.mozilla.com/report/list)
* Supply aggregate information about crashes for this signature
** First date of occurrence / date of spike
** Correlation statistics (OS, plugin, etc)
** Add comments about this signature

=== Feeds ===
* Add RSS feeds for
** New crashes (all and by component)
** Explosive crashes

== Appendices ==
* [https://wiki.mozilla.org/File:Socorro.next_summit.key.gz Summit slides]