View source for DXR Future Work Plan

==What is DXR?==

DXR is a two things.  First, it’s a method for collecting type, member, statement, macro, etc. information about C++, IDL, and soon JavaScript using Mozilla’s static analysis tools (e.g., Dehydra), a hacked build system, and a bunch of scripts.  Second, it’s a web-based tool for mapping this information back onto source code, and allowing users to query and look-up this information.

You can read more about it [http://vocamus.net/dave/?p=557 here].

DXR was written and is being actively developed by David Humphrey.

==DXR Bugs==

* [https://bugzilla.mozilla.org/buglist.cgi?quicksearch=product%3Awebtools+component%3Adxr Existing DXR Bugs]
* Add DXR bugzilla component - [https://bugzilla.mozilla.org/show_bug.cgi?id=517550 bug 517550]
* DXR Tracking bug - [https://bugzilla.mozilla.org/show_bug.cgi?id=489834 bug 489834]
* Deploy DXR at Mozilla - [https://bugzilla.mozilla.org/show_bug.cgi?id=524725 bug 524725]
* Switch from C-based xpidl to Python ([https://bugzilla.mozilla.org/show_bug.cgi?id=498440 bug 498440])

==Current Task List==

The current set of tasks can be summarized into the following high-level categories:

* Improve the DXR web-app backend
* Improve the indexing of source code and creation of the database and HTML
* Improve user access to data stored in the database, making it easier to query, search, etc.
* Improve the presentation and accuracy of data presented within the source code itself

Each of these three have sub-tasks, many of which interrelate.

===Web App Back-end Work===

* Redo existing python to make it easier to extend
* Switch python web cgi to use django
* Audit web app for security
* Start using caching
* Investigate switching from SQLite to a db server
* Investigate adding some basic auditing and user feedback:
** What are people searching for most often?
** Allow users to star or otherwise mark things that are wrong in the HTML data
** Perhaps allow users to leave notes, links, etc. in various parts of the code

===Indexing, HTML Generation===

* Figure out best method for doing continuous re-indexing of the tree
* Improve post-processing speed, investigate better parallelism
* Add proper IDL data to source code such that clicking types, members, etc. shows a popup with links to implementations, callers, etc.
* Fix C++ snippets in IDL to use cpp position indicators (# line file).  Add any other missing IDL features.
** Switch from using xpidl to xpidl.py and add __FILE__ and __LINE__ data to IDL C++ snippets
* Look for speed-ups in how analysis builds are done
* Add links for #include header files
* Add support for C source
* Add support for static functions (see nsContentUtils)
* Add support for JS source.  Consider using jshydra to build static HTML.  See js token info in [http://mxr.mozilla.org/mozilla-central/source/js/src/jsscan.h#60 jsscan.h].
* Add support for Obj-C/C++ source
* Show compiler warnings
* Integrate callgraph: convert to using treehydra, investigate --with-static-checking in mozilla's build sys, include dwitte's callgraph.js and my dxr.js (include "callgraph.js").
* Tokenize macros so that embedded macros are clickable (e.g., NS_ProcessNextEvent resolves to NS_ProcessNextEvent_P)
* Investigate ways to deal with naming collisions.  For example, where types, methods, or macros share the same name, sometimes we pick the wrong one for purposes of linkifying (e.g., NS_ProcessNextEvent).  
* Make IDL pages, and other decl locations do a better job at providing links
* Fix srcdir vs. objdir links (e.g., PRInt32)
* Remove .html from source file names (mod_rewrite?)
* Make dxr scripts path agnostic and more portable (python path stuff)
* Fix ctor html linkifying code so Foo::Foo works properly
* Figure out how to deal with code being ifdef'ed out (e.g., debug vs. non-debug build)
* Audit dehydra script for improvements 
* Treat functions as well as methods
* Add a download-db link
* Link to the hg revision of the indexed db (i.e., a timestamp)
* Add simple hooks and ways for new types of data to be layered into the DB and web app

===Querying, Search===

* Make Users queries differentiate decls/defs
* Add ability to do Users:nsIFrameFrame::GetDocShell
* Add base:type search query
* Show Classname:: as part of member sidebar on the right (for searches)
* Make search results clickable anywhere vs. just on line number and make line number plain text vs. link.
* Make search sidebars links so they can be turned into searches.  For example, members sidebar could be a link that takes you to a member:foo query
* Search UI should encourage using built-in queries, for example, suggesting other keywords (members:) that could be added.
* When searching of nsFoo, display a little summary of the class bases,derived, members
* For Type search results, give a link to do a type:: search to show all members (if there are any) -- nsFoo (class) [members]
* In search results, favour c++ style name ordering (e.g., class nsFoo vs. nsFoo (class) ).  Make the name more prominent.
* In search results, make all types and member names clickable
* Remove side bar from searches, and copy Google Code search results by showing other queries at top of results (Narrow your search: 53 Types, 74 Members, ...)
* In decl/def search results (e.g., member) only show line number once.
* Add counts to all search results (glimpse and otherwise)
* Add pagination to search results
* Limit search nav divs to some lower number (12?) and give a link to see the whole thing.

===Source Code View===

* Pin Popup when scrolling (Jeff and jdm want)
* Make source code nav bar narrow if contents are narrower, and expand to a max size (current size).
* When navigating to search result in code for a type or member, sync the source code navigation pane accordingly
* Lazily keep source code nav and source view in sync.  Clicking any stmt should highlight/scroll into view/etc the containing member/class
* Make source code nav panel sortable (alpha, file order)
* Make types clickable in source code nav bar, perhaps open the popup to show type info
* Make types in source code nav bar always clickable, and add icon to take you to decl (should be able to jump to header file from cpp)
* Add hrefs to links in source code dom nodes on hover
* Standardize on look and feel of source code nav bar for members and popups, etc (gray, icons, font)
* Tighten-up look of popup box
* Fix pop-up position so it isn't off the screen
* Fix Users query so it includes uses of the type directly in addition to uses of derived types (e.g., nsIFoo might have direct users if not an idl type)
* Add Callees and Callers searches to all popups keyed on loc
* Add info to popups that indicates that empty popups are bugs in data/dxr or TODOs (maybe with link to this wiki page or bug filing info).
* Collapse Type Info and Bases tabs in popups
* Convert popup to use fewer/no tabs, and dynamic/tree regions to show data like members, bases, etc.
* Make http://dxr.proximity.on.ca/dxr/mozilla-central/xpcom/ds/nsHashtable.h.html#144/Put work

==Future Ideas (not currently in scope)==

* Integrate other tools:
** Should we integrate [http://www.oxymoronical.com/blog/2010/01/Throwing-xpcom-docs-out-into-the-wild Mossop's XPCOM docs project]?
** Connect with MDC (e.g., for XUL docs)
** Add gcov data, see http://people.mozilla.com/~mnandigama/codecoverage_html/
** Integrate with bug/patch data in bugzilla
** Dead code, should-be-static and other analysis data
* Add Macro decl locations
* Investigate strategy for increment re-indexing vs. full clobbers
* Add support for html/xml, perhaps with links to MDC doc pages
* Multi-platform indexing
** MSVC++ may be doable with this: http://msdn.microsoft.com/en-us/library/87x7wc99.aspx
* Add gcov support
* Add build dependency map info - "what do I have to remake if I change something in content/html/document/src?"
* Add control mechanism for running dxr continuously.  Maybe web based?  Maybe as part of buildbot with waterfall?
* Patch/Review features: try-server type system that checks the patch for issues, but perhaps dxr should be used to visualize the results.  Show things like, new warnings/issues/deadcode.  Also, given a patch, show everything DXR knows about the patch and code affected by it.