DevTools/Features/SourceMap

Feature	Status	ETA	Owner
Source Map	planning	2011-06-01	Kevin Dangoor

Summary

Nearly every web developer starts with JavaScript in one form during development and then produces JavaScript in another form for production use. Most often, this is a case of joining files together and minifying the output to reduce the number of requests the browser needs to make and how much data the browser needs to transfer. Another example of JavaScript starting in one form and ending up in the final form used by the browser is in PHP scripts which will sometimes generate bits of JavaScript as they generate their pages.

When JavaScript gets an error, being able to help the web developer out by showing them where the error occurred in the original source file (PHP script or unminified, still-separate JavaScript for example) would save the developer time every single time they hit an error. The Web Console log messages also include links to the line that generated the log message, so these messages could link back to the original source lines as well.

For a long time, there have been languages that have compiled to JavaScript, and recently there has been a surge in support for these languages led by CoffeeScript. CoffeeScript may seem like a fringe language today, but it will soon be shipping with Ruby on Rails 3.1, giving thousands of programmers a reason for developing with something other than JS.

When developing an application in CoffeeScript, any time an error is hit the browser tells the user the location of the problem in the JavaScript file, not the source file that the user is working with. One new language even went so far as to ensure that the lines numbers in the generated JS match up with the lines numbers in the original source file, which is an unfortunate thing to optimize for.

The solution to this problem is a mapping from the generated JS to the original source files.

Team

• Feature Manager: Kevin Dangoor (irc: kdangoor)
• Lead Developer: Nick Fitzgerald (irc: fitzgen)
• Web Console Contact: David Dahl (irc: ddahl)
• SpiderMonkey Contact: Jason Orendorff (irc: jorendorff)

Release Requirements

• SpiderMonkey needs to output column in addition to source/line.
• When a sourcemap is available, error messages in console will link to the original source, not the generated JS
• Likewise, console log output will link back to the original source
• Patches for CoffeeScript and UglifyJS will be used to produce examples
• A specification for generated scripts to refer to their mappings and a mapping format

Link title

Next Steps & Open Issues

• Decide if starting off without columns from SpiderMonkey is enough of a win to pursue this at this time

• How is the browser notified that a source map exists for a given script?
  • Should it be a comment at the top of the source? Something like //@sourceMap=http://example.com/path/to/source/map
    • This could work with "eval" and script tag injection.
    • John J Barton has suggested that it is best to put the comment at the bottom of a script so that it can be dynamically added without shifting the existing line numbers for a script. I (fitzgen) agree with him.
  • Should it be an attribute on the script's tag? Something like <script src='/path/to/foo.js' data-source-map='path/to/foo.smap'>
    • This would not work with eval, but would work with script tag injection.
  • Should it just be implicit that the source map file is a replacement of the suffix of the script's file? For example, if a script's URL is "foo.js", the source map is implied to be at "foo.smap"?
    • This would not work with eval, but would work with script tag injection.
    • This would result in unnecessary HTTP requests when the source map does not exist.
  • What is th URL pointing to the source map relative to?
    • The script itself, or the page loading the script?

• What requirements must be satisfied before the browser fetches a source map?
  • Should the JavaScript engine be in debug-mode?
  • Should a flag or preference be set?
  • Should the console/firebug/debugger/etc have to flip a switch explicitly when they are started and flip it back off again when they exit?
  • John J Barton points out that the browser shouldn't be handling the fetching of source maps, the tool which is consuming them should (such as the debugger).
    • This neatly side-steps the above issues because if the tool isn't on, it won't ever fetch the source map.

• When should the browser debugger fetch the source map?
  • As soon as it knows of the existence of the source map?
  • By the time a message is logged, or an exception is thrown, it is already too late to fetch the source map because we do not want to delay displaying a message while we go fetch the source map.
    • Incorrect. John J Barton points out that a typical web apps log hundreds of messages and most of them are ignored. We can hold off fetching the source map until the user first requests it by trying to display more information about a particular logged message or error thrown. Until that point in time, the source map information is useless because in all likelihood the user isn't even looking at it.

• When (if at all) should the browser debugger fetch the original source file[s] referenced from a source map?
  • As soon as it has retrieved the source map?
  • John J Barton suggests that it should be delayed as long as possible and only the files which are needed by the user be fetched. Most likely, the user is going to be debugging ~40 or so lines of code, we don't want to fetch all thousands and thousands of lines of code from a large, un-minified app.

• From an API standpoint, how should the browser tell the JS debugger that a source map has been fetched and is available?

• Where are the original source files referenced from the source map?
  • Presumably, they are URLs; what are they relative to?
    • The origin of the script?
    • The origin of the page?
    • I would assume they are relative to the source map (this is what Closure's file format does)

• Where can source maps be located? Do we want to let script's say they come from any arbitrary place?
  • Should they be restricted to the same origin as their script?
  • Should they be restricted to the same origin as the webpage?
  • From either of the above?

• How should the composition of source maps be handled? (e.g. CoffeeScript -> js -> minified js -> concatted with other js files)
  • Should this be the responsibility of the debugger to traverse the source mappings, or should it be assumed that they are composed on the server side?
    • If it is the responsibility of the debugger to traverse the source mappings by hand, then we should list other source map files in the first source map file so that they can all be downloaded in parallel and we don't have to follow a long list of links to other source map files one at a time.

Related Bugs & Dependencies

TBD (status page link)

• Bug 568142: Expand source notes to carry column information - Bug to add line/col information to be available to debuggers.

• Bug 618650: map JS source coordinates to source language that was translated to JS

• Bug 626932: Shrink JSParseNode - Would add character-offset to SpiderMonkey (as opposed to line/col).
  • Has an incomplete patch by njn. Doesn't compile.

• David Flanagan has been working on adding "//@line ###" style comment support to SpiderMonkey which is being tracked by this bug.

Risks

• We could end up with two competing formats if other browsers pursue this at the same time and we're not aware of it.

Designs

The source map file format spec originally proposed by Google. I (fitzgen) have edit permissions and am working with them on making something we can all agree upon.

This feature is related to the Script Origin Tracking proposal, because the mapping should be able to accommodate the many ways in which JavaScript can come into existence on a page.

Test Plans

TBD

Non-goals

• Ideally, the source mapping file will also work for mapping LESS and similar to CSS. However, implementing the user interface for style source mapping is out of scope for this particular feature.

Other Stuff

• Brendan and Jim Blandy have suggested that using a separate mapping file will work better than including the mapping in the generated source.

• A bug with a patch was opened for WebKit but closed INVALID.
  • Webkit has another bug open for this feature.
  • A new bug with a patch was opened.