line number mapping for debug

Question

line number mapping for debug

pmuellr opened this issue 14 years ago · 181 comments

There's been some email chit-chat regarding support coffeescript in the existing browser debuggers. The basic idea is to include information in the generated JavaScript files that the debuggers can make use of to display the original coffeescript source, and deal with line mapping issues between the CoffeeScript source and generated JavaScript.

Thought I'd go ahead and open up an issue for discussion.

Answer 1 · 2010-07-30T16:45:48.000Z

So some thoughts:

CoffeeScript source location - somehow we need the coffeescript source. See Issue #557 as one possible solution. The debugger would then be responsible for finding the source given that "URL". Java current just uses the basename() of the file as the file name in it's debug info, but of course it also has the full class name available to it as well. Not sure how we might get some "relative" name in there, so it seems like basename() or abspath() of the input file are the choices. abspath() exposes more information than it should, basename() will cause naming conflicts for same-named files in different directories
line number information - The debugger will eventually want this in hash tables, both directions. Go from a cs line # to js line # when setting breakpoints, and js line # to cs line # when the JS vm is communicating to the debugger (eg, "stopped at line 47"). Presumably the actual data for the line number data could be in JSON format (eg, an object literal mapping cs #'s to an array of js #'s).

Answer 2 · 2010-08-06T04:50:03.000Z

Might be worth looking at this also:
http://nex-3.com/posts/92-firesass-bridges-the-gap-between-sass-and-firebug

Maybe this same type of strategy could be used so that firebug could step through javascript line by line, but display the coffeescript code instead.

Tough project, but could turn into something pretty killer!

Answer 3 · 2010-08-17T23:19:30.000Z

I'd be glad to help someone out with this ticket, but I'm not going to be spearheading it myself ... so closing as frozen for the time being. If someone wants to step up and do a port of FireSass for CoffeeScript, do let us know.

Answer 4 · 2010-12-12T13:56:52.000Z

Alright -- this ticket now has life again, in Firefox at least:

https://bugzilla.mozilla.org/show_bug.cgi?id=618650

Reopening...

Answer 5 · 2010-12-13T01:10:26.000Z

Quick helpful workaround:

Include each line of the original coffeescript source inside a block comment before the js code that it generated. You could include the original coffeescript source lune numbers in the comments.

That would at least help simplify the learning and debugging process.

Answer 6 · 2010-12-13T11:55:27.000Z

Mascara has a source-line mapping utility that only works in Firefox. It's quite possible to implement something similar:

try
  console.log 'Hello'
  a for i in [0..10]
catch error
  lines = { 31: 3, 32: 3, 33: 3 }
  error.message = error.message + ' on line ' + lines[error.stack.split('\n').shift().split(':').pop()]
  throw error

Run on Try CoffeeScript. Firefox only, though.

Answer 7 · 2010-12-18T19:44:44.000Z

Related code:

Closure Compiler (JS -> JS)

...can generate source-to-source mappings:

java -jar $CLOSURE_JAR --js example.js --create_source_map ./map --js_output_file compiled.js

SourceMapLegacy.java has comments describing the format.

_PYXC-PJ (PY -> JS)_

...can generate mappings using the same format, but with more file info: [{"sha1":"...",...}] instead of []

commit: "changed emit() return type from a string to list of strings and kids"

commit: "source-to-source mappings! (same format as Closure Compiler)"

** exception-server **

An exception logging/viewing server has been written (in NodeJS) that uses these mappings to convert an exception's stack information into pre-compilation code snippets with TextMate URLs.

(your compilers/scripts send the server (code_sha1, mapping)s and (sha1, code)s at compile-time)

I'll post it [REDACTED TIMEFRAME] when it's more documented/polished.

** common-exception **

...can extract file/line/[col] information from NodeJS, Firefox, Chrome, Safari

Answer 8 · 2011-01-05T22:27:36.000Z

Crazy idea: What if when the CoffeeScript was compiled to JavaScript, it was done such that the lines corresponded 1:1 by placing multi-line JavaScript conversions on a single line.

Answer 9 · 2011-01-05T23:18:15.000Z

I suspect it is crazy, but it would be useful. I actually do that with https://github.com/pmuellr/scooj , but only because the source is largely unprocessed JavaScript anyway. The nice thing is that even though you see some grungy code in the debugger, you do know the exact line number to go to in your editor.

Answer 10 · 2011-01-06T23:53:35.000Z

I suggest a --debug coffee parameter that does something like this:

# Add Core-Utils to Underscore Namespace
_.ducktype = (obj, methods...) ->
  some("code")

to become:

var __slice = Array.prototype.slice;
// Line 2
_.ducktype = function() {
  var methods, obj;
  obj = arguments[0], methods = 2 <= arguments.length ? __slice.call(arguments, 1) : [];
  // Line 3
  return some("code");
};

this would work Browser independent and i think(im not a coffescript compiler expert, though) it could be implement without big hassles.

Answer 11 · 2011-01-07T18:10:26.000Z

Let's define some terms:

Def a "compiler" compiles a non-empty set of "source file"s to exactly one "compiled file"

Defs

a "(coord to coord)-mapping" is a function from (line, col) in the compiled file to (source_file_ref, line, col)
a "(line to line)-mapping" is a function from (line) in the compiled file to (source_file_ref, line)
a compiled file is "{coord,line}-{coord,line}-commented" if it contains comments in some format which provide a ({coord,line} to {coord,line})-mapping
a "(...mapping...) file" is a byte-string in some format which contains a representation of a (...mapping...)

Notes:

Don't worry, the representations of these mappings are usually much more space-efficient than the ultra-trivial {coord1:info1, ...}
source_file_ref should include sha1(source_file) if possible, because that's really useful when used with my not-yet-released exception server

Defs

"CCMF" is a (coord to coord)-mapping format. It's what Closure Compiler generates
"CCMFH" is a backward-compatible extension with required hashes and optional other info

See my writeup of CCMF[H]: https://gist.github.com/769845

Answer 12 · 2011-01-07T19:37:05.000Z

.lineno

Recall: adding .lineno to each node instance is a solved problem:

#955

With no side effects, that grammar.coffee change could be merged at any time.

Answer 13 · 2011-01-07T19:37:45.000Z

Let's get this done! (Please? Please? Please?)

CoffeeScript should support both

{coord,line}-line-commenting in whatever that format turns out to be (e.g. via --line-commenting)
exporting CCMFH files (e.g. via --ccmfh-out=)

(1) might need to wait for some format finalization, but (2) does not.

And the nodes.coffee work required for (2) will apply to (1).

I'm trying to do a FOSS triple-launch in [REDACTED TIMEFRAME]:

a NodeJS web framework
a coherent Freedom Box attempt
an exception server with awesome mapping support and TextMate links to your uncompiled source files
- Screenshot of a mapping-following representation of an exception that occured while NodeJS was running code compiled by PYXC-PJ from Python syntax

(all three are written in CoffeeScript)

...and it would be really, really awesome to have CoffeeScript CCMFH support before then so I can integrate it.

Imagine being able to hit Command-R and have jashkenas/coffee-script-tmbundle compile, invoke node, and then represent the resulting exception in a beautiful and convenient mapping-following way with CoffeeScript source links and code snippets...

Answer 14 · 2011-01-07T19:39:08.000Z

Some modest proposals

The Good (but more complex and possibly slower)

Something like what PYXC-PJ does.

EDIT: e.g.

Throw
  compileNode: (o) ->
    ...
    something(o, ["throw ", @expression, ";"])

Instead of having the main compilation function return a string, have it return a BAR.

BAR ::= ELEMENT-list

ELEMENT ::= string | FOO | BAR

FOO ::= something which can
    1. provide the code fragment for that subtree
    2. provide the ((line, col) -> (source line))
       mapping information for that fragment

Some function can then use the resulting BAR to generate both

the usual JavaScript
a CCMFH file

PYXC-PJ commits:

I'm not a nodes.coffee expert (yet) so I'm not sure if subnode.compile(o) can be deferred. If so would let FOO be, e.g., {subnode:@expression, o:o}

Would o have been destructively modified by the time foo.subnode(foo.o) gets run?

I think The Good could be done without deferring .compile, but that might be harder.

The Bad (but simple (though verbose) to implement and possibly faster)

Something like what Closure Compiler does.

Instead of the beatifully concise current compileNode methods, do something like:

    something.startNode this
    something.add           "throw {"
    something.addSubnode    @expression, o
    
    # addSubnode: (node, o) ->
    #   ...
    #   node.compile o
    #   ...
    
    something.add           ";"
    something.endNode()

This would invoke the .compiles in the same order as the current implementation.

The Fugly

Have whatever methods invoke compileNode replace ([...].compileNode([...])) with

((() ->
    token = newUniqueToken()
    (
        "/*FUGLY:#{token}:@lineno*/" + 
        [...].compileNode([...]) +
        "/*/FUGLY:#{token}*/"
    )
)())

...and write a crazy function that uses the resulting text monstrosity to generate (comment-free JS, a CCMFH file)

This would be totally insane, but it would work and it would involve the least modification to nodes.coffee

Answer 15 · 2011-01-07T20:35:17.000Z

I'll work on a

something(o, ["throw ", @expression, ";"])

approach

Answer 16 · 2011-01-09T15:21:54.000Z

BAR-ification of nodes.coffee in progress:

https://github.com/andrewschaaf/coffee-script/commits/master

After each change, I run

git checkout -- lib && cake build && cake build && cake test && git status

and confirm that the tests pass and that no lib/*.js files other than nodes.coffee are changed

Answer 17 · 2011-01-09T15:24:55.000Z

From #coffeescript yesterday:

my point about a codegen method's "//#{@child.compile o}" vs "#{@child.compile o}//" was that the codegen method currently discards the information distinguishing the two.

the relativeCharOffset of each kid matters.

one option would be for the codegen method to store that information in its kids

e.g. for Throw ("throw #{@expression...};"),

@expression.relativeCharOffset = "throw ".length

but this could make the codegen methods uglier than with the BAR-returning approach

with the BAR-returning approach, the Base method calling the codegen method can process the BAR immediately, computing the flattenedCodeString as usual and (processing the mapping implications if mapping:true)

BAR-returning codegen methods are a way to abstract out the mapping-handling code

from the outside, you're still calling the top node's .compile or .compileWithMapping

Answer 18 · 2011-01-09T15:24:58.000Z

From #coffeescript yesterday:

my point about a codegen method's "//#{@child.compile o}" vs "#{@child.compile o}//" was that the codegen method currently discards the information distinguishing the two.

the relativeCharOffset of each kid matters.

one option would be for the codegen method to store that information in its kids

e.g. for Throw ("throw #{@expression...};"),

@expression.relativeCharOffset = "throw ".length

but this could make the codegen methods uglier than with the BAR-returning approach

with the BAR-returning approach, the Base method calling the codegen method can process the BAR immediately, computing the flattenedCodeString as usual and (processing the mapping implications if mapping:true)

BAR-returning codegen methods are a way to abstract out the mapping-handling code

from the outside, you're still calling the top node's .compile or .compileWithMapping

Answer 19 · 2011-01-13T23:14:10.000Z

About ten years ago I wrote som crude source-to-source compilers for Perl and C++ which I called Pyrl and Cython. They simply gave you pythonish indent-syntax. They had the flag -p for pretty output and some other for line-to-line, which esentially means that it simply made sure that lines in the output ended up in the same line number as in the input, no matter if certain lines became 'ugly' because of code stacking.

Simple solution. Works in all environments (like Jake that compiles the coffee-makefile from inside it's script), any editor, etc., shouldn't be to hard to implement?

Answer 20 · 2011-01-28T21:19:19.000Z

How hard would it be to add an option to have the js output include a call to a custom trace function on every function call? The param could be the callee function name and optionally the original line number and file. This would really help debug those cases where the code doesn't finish because I forgot to call something else at the end of a callback. These cases drive me crazy and cause me to put log calls all over the place.

The convention could be that if a specific function exists at the top level then the feature is enabled and that function is called.

debugFnCallLog = (funcName, fileName, lineNum) ->
    myCustomLogRoutine ...

Answer 21 · 2011-04-19T08:43:49.000Z

For reference, here is a related webkit ticket: https://bugs.webkit.org/show_bug.cgi?id=30933

Answer 22 · 2011-04-19T19:03:26.000Z

In my fantasy world V8 and other engines would be broken into a separate parser and byte-code interpreter while retaining the same JS semantics.

Answer 23 · 2011-05-03T22:31:02.000Z

Just want to toss out a major +1 to preserving line numbers, i.e. mapping 1:1 from source Coffee line # to generated JS line #. The excellent Streamline.js library has this option for its transformations, and it has proved invaluable.

Btw, if it helps, our platform is Node. We too compile dynamically w/out generating JS files, so things like line-numbers-in-comments aren't helpful. Our use case also isn't e.g. Firebug debugging generally but more "my program threw an error on line 512, where is that?"

Thanks in advance! Excitedly looking forward to seeing what comes of this.

Answer 24 · 2011-05-04T14:22:11.000Z

+1 for this

Answer 25 · 2011-05-04T14:27:00.000Z

Anyone in the NYC area interested in meeting up for a line-mapping mini-hackathon on Sunday the 15th?

Answer 26 · 2011-05-04T23:43:37.000Z

No, but I'd personally donate beer money. This needs to be done.

On May 4, 2011, at 10:27 AM, andrewschaafreply@reply.github.com wrote:

Anyone in the NYC area interested in meeting up for a line-mapping mini-hackathon on Sunday the 15th?

Reply to this email directly or view it on GitHub:
#558 (comment)

Answer 27 · 2011-05-05T01:14:02.000Z

+1 I'd send some coffee + beer money too for this, it would be the biggest evolution for CoffeeScript

Answer 28 · 2011-05-05T01:17:40.000Z

I'm surprised that no one has ever implemented one of the several quick
kludges that have been suggested.

On Wed, May 4, 2011 at 6:14 PM, tanepiper <
reply@reply.github.com>wrote:

+1 I'd send some coffee + beer money too for this, it would be the biggest
evolution for CoffeeScript

Reply to this email directly or view it on GitHub:
#558 (comment)

Mark Hahn
Website Manager
mark@boutiquing.com
949-229-1012

Answer 29 · 2011-05-05T13:08:52.000Z

I'll actually be in the NYC area on the 15th with nothing to do. I may be up for this.

Answer 30 · 2011-05-06T14:15:26.000Z

+1 for some way of debugging coffeescript. It would be really helpful.

Answer 31 · 2011-05-06T14:58:53.000Z

The 15th is a go. [EDIT: multiple parties cancelled.]

Possible attack plan:

Add .line_number to nodes (satyr's implementation involves more changes but seems cleaner and safer than this hack)
Create mapping files via fugly hack (see above)
NodeJS exception logging/viewing server (to run on localhost) with {mapping file support, code snippets, TextMate links}
Create mapping files properly

Answer 32 · 2011-05-09T16:18:30.000Z

I definitely vote for line to line compiling without mapping files. Way to go! :-) If I wasn't in 'way after deadline'-mode on my current project, I'd help out right away.

Answer 33 · 2011-05-10T16:17:03.000Z

This WebKit / Web Inspector bug may be of interest to folks:

Bug 60550: Web Inspector: add protocol method for loading script source mapping

Presumably for Traceur (it's from Google folks), and just describes the "protocol" by which source mappings are obtained, not the format of the source mappings of themselves.

Answer 34 · 2011-05-15T23:16:56.000Z

@michaelficarra and @andrewschaaf -- somehow missed you two were working on a solution to this together (today?). I got somewhere with it if it helps you! https://gist.github.com/973659
Edit: Hah! @andrewschaaf, missed everything after early May before working on this -- came to many of the same solutions you did :) Wish I'd seen it sooner!

Answer 35 · 2011-05-16T19:04:15.000Z

@geraldalewis: Sweet!

@michaelficarra and I were too busy that day to meet up and hack. I made some progress on a line-mapping-friendly exception-server. I'll get it done and posted [redacted].

Answer 36 · 2011-05-16T19:06:40.000Z

With this many comments on a GitHub page, we need to start posting some 9f09aeb-style lulz...

Answer 37 · 2011-07-07T17:06:13.000Z

Web Inspector: draft implementation of compiler source mappings support in debugger:

https://bugs.webkit.org/show_bug.cgi?id=63940

Draft description of source maps for Web Inspector (linked to in the bug):

http://bit.ly/n4YaTz

The actual format of the sourcemap is not yet specified, though it's presumably JSON as the bug talks about retrieving the source maps via JSONP.

The interface to be provided based on the data is not completely specified, but there's bits of it here, in the section for new source file "CompilerSourceMappings.js":

https://bugs.webkit.org/attachment.cgi?id=99713&action=prettypatch

The idea is, we have the raw source map data, and then provide a JS object which matches that shape that use the data to return the expected results, and voilà! CoffeeScript debugging in Web Inspector.

Answer 38 · 2011-07-07T17:24:06.000Z

pmuellr: When will that be live?

I would not mind the one-to-one idea in the mean time:
"Crazy idea: What if when the CoffeeScript was compiled to JavaScript, it was done such that the lines corresponded 1:1 by placing multi-line JavaScript conversions on a single line." - lucaswoj

Answer 39 · 2011-07-07T17:34:06.000Z

treeform: no telling when it will be "live". Work in progress. That could be committed tomorrow, say, or never. Good news is that it's being done by the GOOGlers who are the primary Web Inspector devs, so it will likely ship, sometime. Would first be available in WebKit nightlies, so might want to install that now so you can be ready the minute it makes it into the build! cc yourself on the bug, of course.

I'd rather see any work done, be in generating "source maps", whatever form that may be. It's the answer long-term, compared to the 1-1 line # story. Easy for me to say tho, since I'm not doing ANY of this work ... :-)

Answer 40 · 2011-07-14T17:16:12.000Z

It appears that the Moz and Google Closure folks are now also on the case! Here's another Source Map proposal linked to in WebKit bug 63940.

Answer 41 · 2011-07-14T17:20:44.000Z

Speaking of formats, here's what Closure Compiler uses now:

https://gist.github.com/1082926#file_foo.simple.js.mapping

(it's a newer format than what they were using earlier in this thread...)

Answer 42 · 2011-08-09T18:46:41.000Z

Hello! I am a summer intern for Mozilla acting as the lead on the source map project. The source map project will allow mapping any X to JS compilation's line/col/source location info including among others CoffeeScript to JS and JS to Minified JS as long as there exists a source map file. Here's a quick overview of the project status:

I have implemented the core library for parsing and generating source map files (the spec which describes what Mozilla, Closure Compiler, and (if I'm not mistaken) Webkit will support is here https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k/edit?pli=1) - https://bugzilla.mozilla.org/show_bug.cgi?id=669999 and https://github.com/mozilla/source-map
I have implemented the bulk of the integration between the webconsole and source maps WRT logged messages and uncaught errors - https://bugzilla.mozilla.org/show_bug.cgi?id=670002
The feature page which tracks the source map project is located here: https://wiki.mozilla.org/DevTools/Features/SourceMap

Now I'd like to begin working source map generation in to CoffeeScript. I'll take a look at previous patches and see if I can't reuse a bunch of that code.

So yeah, I just wanted to say hello and publicly announce my intents.

Answer 43 · 2011-08-09T20:34:21.000Z

@fitzgen: awesome. I'm sure I'm not the only one here who's hugely excited about this ^_^

Answer 44 · 2011-08-10T02:52:37.000Z

@fitzgen: from the source map proposal:

Additional fields may be added to the top level source map provided the fields begin with the “x_” naming convention.

That sounds like a really bad design decision. Why not nest objects under some extensions property?

edit: Sorry, I meant to bring this up a few weeks ago, but I couldn't add comments to the google document, so I just kind of forgot about it.

Answer 45 · 2011-08-10T03:53:04.000Z

@fitzgen Awesome!

I agree with @michaelficarra – an extensions property seems a better choice here.

Answer 46 · 2011-08-10T08:20:39.000Z

+1 for @michealficarra's idea.

Answer 47 · 2011-09-23T02:42:38.000Z

What is the status on adding lines/columns to the Node objects?

Answer 48 · 2011-09-23T06:31:48.000Z

@showell, The commit above should maintain the line/col info in the AST after parsing is finished. Unfortunately, I haven't finished the second half where I actually use that information to generate source maps. I have a bunch of dirty stuff that needs to be revisited and probably rethought, but at the moment I am preoccupied with other things.

Answer 49 · 2011-09-23T11:34:18.000Z

We'll also need a VLQ library, to read/write integers.

There was also some discussion in webkit bug 60969 concerning
how Web Inspector will retrieve the sourcemap files. There are options:

JSONP via script injection
JSON via CORS (claims that it won't work though)
IFRAME html wrapper that access the files

We may need to be generate JSONP wrappers for the sourcemap/source, or an extra .html file, or something.

Answer 50 · 2011-09-23T15:14:40.000Z

@pmuellr, the spec already defines how a source map should be fetched, see "Linking generated code to source maps" in https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k/edit?pli=1

Also, my core source map lib, which is meant to be used by Firefox internals, compilers and minifiers, developer tools, etc already handles the VLQ stuff as well as actually generating and consuming source maps as long as you can pass it the information required.

https://github.com/mozilla/source-map

Basically, all that needs to happen to make CoffeeScript generate source maps is to replace all the string interpolation and concatenation with building a tree of SourceNodes which are part of that core library above. Each mapping in a source map needs the original line/col, original source, and generated line/col. Using the SourceNodes just allows us to maintain the original line/col/source info while building up snippets of generated JS. Then when the tree is complete, we can do an in order traversal of the tree as we concatenate the snippets, and by doing this we will always know the generated line/col for a given location while we concatenate and we will have all the info we need.

Little bit of a brain dump, if it doesn't make sense I can try and clarify.

Answer 51 · 2011-09-23T16:15:30.000Z

All the spec mentions w/r/t "Linking generated code to source maps" is how to "link" to the source map in the generated source. It doesn't specify, nor should it, how a debug agent actually GETs those files, once it has the links. If you were implementing a debugger as a plain old html app (say, like weinre or running Web Inspector in "remote" mode) how would you go about actually GETting the files? That's the nub of the issue. Presumably you'd use XHR or whatever. Then you may have cross-origin issues. Etc. Again, see webkit bug 60969 for a discussion of the techniques being considered.

This is a rubber-meeting-the-road issue. How do we actually hook this all up and make it work?

Happy to see the https://github.com/mozilla/source-map stuff. Except it's MPL/GPL/LGPL so some folks will have issues. Like the coffee-script project. 👎

How are folks supposed to contribute to the "discussion" in the referenced Google Doc? Is the discussion happening in the doc, or somewhere else, and if it's in the doc, how do I get r/w access to it? This isn't a great place to discuss this stuff.

Answer 52 · 2011-09-26T23:12:18.000Z

So is Google talking about adding support for Source Map (or something similar) to v8? This would help out when using CoffeeScript within node.js.

Answer 53 · 2011-10-05T01:27:23.000Z

I am asking for this one-line patch to be accepted for src/grammar.coffee:

-  [patternString, "$$ = #{action};", options]
+  [patternString, "$$ = (function(){ var _ = #{action}; _.lineno = yylineno; return _; })()", options]

I totally understand that browsers are still working out line mappings, but this patch has applicability that extends beyond those use cases, and I can't imagine a more minimal patch.

Answer 54 · 2011-10-05T01:39:26.000Z

@showell: open a pull request with some use cases.

update: the next series of comments will seem out of place, since @showell has deleted his comments

Answer 55 · 2011-10-05T03:08:51.000Z

Yep, a minimally-invasive way to get line numbers in to the AST would be great.

Answer 56 · 2011-10-05T03:56:17.000Z

@showell:

it gives us a place to discuss it -- now and in the future when someone asks about it
it makes it easier for us to merge
we can see it in the proper context
why bitch about it when it's a means for you to get what you want?

Answer 57 · 2011-10-05T04:06:00.000Z

@showell -- that line looks like it's from my patch in #1396 :

[patternString, "$$ = (function(){ var _ = #{action}; if(typeof _.lineno !== 'undefined'){_.lineno = yylineno; }; return _; })()", options]

(No idea why I chose _ instead of a descriptive name, but that tags each node with its line number within Jison.)

I'm not sure if that line can be made more elegant, though I hope so. The cleanest solution would be a patch for Jison, but that creates new problems. I prefer this approach over modifying the grammar (the strategy that @fitzgen employed in 7592664 and @satyr uses in coco). Modifying the grammar to include numbers feels creaky and conflates grammar with metadata unnecessarily.

I think you'll still need to elucidate your use cases as @michaelficarra suggested :)

Answer 58 · 2011-10-05T04:41:09.000Z

Note that:

yylineno refers to the last token in the rule, so attaching it blindly can screw up.
creating 200-ish IIFEs within one switch has certain overhead.
we'll have to adopt @fitzgen's approach for source mapping eventually.

Answer 59 · 2011-10-05T04:55:47.000Z

@showell: In English, "bitch" used in that context is not a derogatory term. It means "to whine or complain". I was asking why you would rather complain about opening a pull request than just do it and have us formally consider it. Pull requests are a nice standard practice with many benefits that I listed above. For some reason, you (and a few others like you that I've interacted with here on github) would rather attack me in a knee-jerk reaction than consider the fact that you're interpreting my words incorrectly. I was just trying to help you.

the "patch" you submitted would be a lot easier to merge with one simple click of github's merge button
its proper forum would be an issue (pull request) dedicated to it
again, it would be helpful to evaluate the patch when given the context in the pull request
our compiler doesn't need to add line number information to the AST; it works perfectly fine without it
- yes, it would be more interoperable
- but I also couldn't care less about it now that I see you don't really care about how the other people that work on the project are treated

edit: minor wording/punctuation

Answer 60 · 2011-10-05T05:21:13.000Z

@showell:

Yes, it's a very similar amount of work, but only once we've already determined that it
1. is one of our goals (it probably would be)
2. implements it the way we want (very unsure)
3. performs well and generates nice output
4. a whole laundry list of other approval conditions
This isn't the proper forum; this issue is about line number mapping: it's pretty much about source maps now
I meant the context around that line of code. We can make more informed decisions when that's just presented to us.
I'm not calling for inaction. All I requested is that you create a dedicated place to continue discussion of your proposal. It wasn't unreasonable. It would have taken you only a few minutes. That's why I questioned you complaining about it. It made it seem like you didn't really care that much.

Answer 61 · 2011-10-05T05:24:53.000Z

@satyr Can you elaborate on making yylineno more precise?

You can't really make it any more precise--the logic is hard-coded into the Jison generated parser.

good to know the edge cases

e.g. https://github.com/jashkenas/coffee-script/blob/1.1.2/src/grammar.coffee#L492

Can you elaborate on your other points as well?

The rules are compiled into one big switch statement.
- See parser.js.
- Try cake bench (for perf) and cake build:browser (for size).
yylineno will be unused when adopting source mapping (as we'll need column numbers as well). See @fitzgen's patch.

Answer 62 · 2011-10-05T05:51:45.000Z

If you are arguing that my patch is harmful, I'll put up more of a fight, but I'll defer to reason

As always, I'm:

merely pointing out the patch's implications.
not the one to decide. It's up to Jeremy.

Answer 63 · 2011-10-05T06:21:07.000Z

Guys, can you please chill out? At time of writing there's 24 people subscribed to this issue, many of whom are probably getting email notifications like me. I don't know about you, but I could sure use less internet-grade arguments in my mailbox. This isn't the place for a personal spat.

Can we please get over who's "right" or who "started it", open a new pull request, and forget about the whole thing?

Can we just move on with our lives?

Answer 64 · 2011-10-05T15:37:25.000Z

Sincere apologies for my stupid comments. I went back and deleted a bunch of them, so I apologize if I have now put some other's folks comments out of context.

This is the proposed patch:

-  [patternString, "$$ = #{action};", options]
+  [patternString, "$$ = (function(){ var _ = #{action}; _.lineno = yylineno; return _; })()", options]

It's mostly a small excerpt from a more recent patch from @geradlewis. It tags each node with its line number within Jison. Perhaps a more descriptive name than "_" could be used. I will open a new pull request soon.

Answer 65 · 2011-10-05T21:45:07.000Z

Happy to see the https://github.com/mozilla/source-map stuff. Except it's MPL/GPL/LGPL so some folks will have issues. Like the coffee-script project. 👎

When writing that lib I just used the conventional Mozilla tri-license without really thinking. I've opened up a bug to get it re-licensed to BSD so that it is easier to include in projects like CoffeeScript: https://bugzilla.mozilla.org/show_bug.cgi?id=692281

Answer 66 · 2011-11-02T13:25:04.000Z

Anyone in NYC (and globally) up for a #558-and-related-tools hackfest all day Saturday Nov 26?

http://www.meetup.com/nodejs/events/39626642/

Answer 67 · 2011-11-07T14:14:06.000Z

I haven't been able to read through all this, but couldn't a debug mode just output a comment line of the CS code that's being converted to JS right before the conversion?

//math =
math = {
//root: Math.sqrt
root: Math.sqrt,
//square: square
square: square,
//cube: (x) -> x * square x
cube: function(x) {
return x * square(x);
}
}

Sure... it'll probably double your JS size... but it's a debug.

Answer 68 · 2011-11-07T14:32:56.000Z

@sevifives Your suggestion seems pragmatic, but the root causes of CS not having line numbers are twofold:

All the patches that have been submitted have either been ignored or deemed too complex. In fairness to Jeremy, he's a busy guy, and there are indeed bigger fish to fry.
The current main-branch implementation discards line numbers at the lexing stage. When you discard line numbers at the lexing stage, even your seemingly pragmatic suggestion becomes difficult.

I have a fork of CS that implements line numbers:

https://github.com/showell/coffee-script/commit/4961119e541c24f1ca744b38cab46ee656c245f4

I haven't bothered to package up the pull request yet, because it doesn't seem to be a high priority, but the code is there for the taking.

So far my patch only goes as far as passing line numbers to the Node instances, and you can get line numbers in the --nodes output. It would be a fairly simple matter to extend my patch write a source code mapping. At least one other person has already done this, but his patch was a little messy in its strategy for passing line numbers into the node objects.

There is nothing technically difficult about providing line number support in a transcompilation language; it's just a weakness of the current implementation.

Answer 69 · 2011-11-07T15:13:45.000Z

It's not just about debug mode.

It's important to be able to go from a browser-side exception stacktrace and have your exception logging/viewing system follow the line maps for Closure Compiler minification, then coffeescript, then coffeesurgeon bodystitch-ing or connect-assets snocket-ing all the way back to specific positions in your repo's specific commit's .coffee coffee files.

Join the hackfest and get your #558 on! 3 RSVPs so far.

Answer 70 · 2011-11-16T17:30:34.000Z

#558 hackfest update:

The date is now Saturday, December 10th.

The page is now under the CoffeeScript New York meetup.

http://www.meetup.com/Coffee-Script-New-York/events/41391392/

See you there!

Answer 71 · 2011-12-07T15:54:30.000Z

See also: 1918

Answer 72 · 2011-12-09T19:44:50.000Z

@andrewschaaf and others, I have written up some notes that might help folks get up to speed on line number support:

https://gist.github.com/1452964

My approach is hacky, but it requires very minimal code changes, so it might be useful for a warmup spike in the hackfest.

I wrote it up as a gist, instead of a patch, because some of the code is basically scaffolding, but if you go to the "showell" branch of coffee-script, there's a working version that includes all the changes from the gist.

Answer 73 · 2011-12-09T20:50:23.000Z

@showell: Oof. That is really non-trivial, isn't it...

Answer 74 · 2011-12-10T11:54:17.000Z

andrewschaaf commented 13 years ago

Answer 75 · 2011-12-10T22:46:39.000Z

@jimtla and I have proof-of-concepts for all of the above except ugly -> (js, map): https://github.com/jfdi

Answer 76 · 2011-12-20T02:39:27.000Z

Hey line-number troopers.

I just got an email update from @fitzgen on the current state of Source Maps in Firefox, and I hope he doesn't mind if I share part of it with y'all here:

Unfortunately, I think the work has slowed down a bit. [...] The state of
things where I left them was that there were a few things that depended
on the new JS debugger API which has been in the works for some
time now, and that my source map work couldn't land until that code
was finished. You can look at the dependency graph between all the
bugs and perhaps get a better idea of what I am talking about:
https://bugzilla.mozilla.org/showdependencygraph.cgi?id=670002

I expect that the patch to the webconsole which is attached to bug
670002 needs to be updated as well because the webconsole wasn't
e10s (seperate processes for chrome and content) compliant at the
time that I wrote the patch, and I believe it is now.

To sum things up, I think there is a little bit of plumbing left to do
between the debugger and the webconsole's source map integration,
as well as checking patches for bit rot. The biggest thing is the new
debugger features that the source map integration relies on, and I am
not sure when they will be ready to land.

Answer 77 · 2011-12-20T06:36:02.000Z

Guys, I realize that most people probably use CoffeeScript in the browser, but waiting until Firefox supports Source Maps isn't very satisfactory to me. 1) Everyone I know uses Chrome for development and 2) Source Maps wouldn't do anything for people who are running CS against node.js. I appreciate everyone's work on this, but it seems like trying to come up with a complete solution (e.g. @andrewschaaf's three-fold strategy) isn't completely necessary, at first. I know this question has been asked before, but is it not possible to simply tag the generated JS with line numbers? This would get 90% of the way there for people who simply want a way to map JS back to CS. The other 10%, such as mapping minified JS to unminified JS -- that sounds pretty neat, but consider the fact that no JS compiler has this capability at the moment, so no one is clamoring that this particular problem be solved. You know?

Answer 78 · 2011-12-20T13:08:55.000Z

@mcmire: You're right. Waiting for browsers tends to be a fool's game ;)

Tagging the generated JS with line numbers may be wise -- since this ticket was first opened, lack of line numbers has become the favorite whipping boy of folks looking for a reason to ignore CoffeeScript. Even if we don't tend to use it, perhaps --line-numbers would make for a nice compiler option.

Answer 79 · 2011-12-20T13:52:45.000Z

Hackfest summary:

only 10% showed up
@jimtla and I got the above slide 80% done @ github.com/jfdi (thanks for the headstart, @showell!)

We need to get the second 80% (tests, visualization tools, correctness...) done some other weekend. Jan 7, perhaps?

--line-numbers would be nice, and it would get line numbers into the AST on master.

"no JS compiler has this capability at the moment"

Closure Compiler's --create_source_map and com.google.debugging.sourcemap
PYXC-PJ for about a year

"no one is clamoring that this particular problem be solved"

And 1800s focus groups would have found that people wanted faster horses.

Answer 80 · 2011-12-20T14:14:24.000Z

"no one is clamoring that this particular problem be solved"

26 on this thread, and it's probably one of the most common complaints I hear about CoffeeScript

Answer 81 · 2011-12-20T17:17:44.000Z

@jashkenas @mcmire @andrewschaaf @tanepiper Apologies in advance that this is mostly +1 to what already has been said, but, yep, I totally agree with a couple ideas that have been expressed recently on this thread.

Line number support is a major sticking point for CS adoption. Most of the stick-to-JS rants you see on the web are kind of silly, but they almost all mention debugging as a sore point, and it's impossible to refute their current concerns, even if they exaggerate the difficulty and/or ignore the inevitability of progress.
We can add plenty of debugging improvements to CS that have nothing to do with browser source maps. I actually write as much CS on the server side as in the browser. Even for browser code, some kind of line number annotation would make debugging easier, even before any support comes from the browser inspectors.

I've poked around with line number support, and here are the decision points that I see as important:

External source map or annotated JS? Simply annotating the JS has a certain simplicity, but I think an external source map is ultimately more flexible, so that third party tools don't have to wade through the generated JS to get line number mappings, and so that the JS itself stays small.
Parser cleanup? There have already been a few patches submitted for line number support, and they have tended to be fairly invasive. Part of the problem is that there's not a one-to-one relationship between grammar elements and AST nodes. There are good reasons that this is the case, but the impedance mismatch makes it challenging to patch in line number support without a lot of special cases.
Roadmap? If we simply attached line numbers to the AST objects, then third party tools could access the line numbers even without a special command-line option. This would be a great start, but it wouldn't help actual debugging until JS line numbers were married to the CS line numbers. It would, on the other hand, allow for tools like linters and visualizers to be written.

The biggest sticking point is probably the parsing roadmap. If there is any intention to rearchitect grammar.coffee, then that should happen before any serious line number efforts. I know @michaelficarra has ideas in this regard, and I suspect @satyr would have insight as well. Has anybody been watching this project?: https://github.com/fab13n/parsec-coffee-script

Answer 82 · 2011-12-20T20:51:35.000Z

The Chrome/WebKit/Web Inspector folks are also working on SourceMaps, as well as the FIrefox folks. I don't know what the current status is though.

+1 on --line-numbers:

it will shut people up, a little
it will force the mapping of original line # to generated source into the CS translator (assuming it's not already there), which will be nice infrastructure to have in place once real SourceMap (or whatever) support comes along

I'll also note that I don't believe there is any SourceMap support working anywhere, for anything, in the browsers. It isn't fully baked. Who knows, maybe it'll never work. I wouldn't suggest start working on actual SourceMap support until one of the browser vendors show it working on something - a dumb minizer or something.

Answer 83 · 2011-12-22T05:25:03.000Z

I've started work on a patch to provide source maps (or line numbers, once you can do the first the second is easy)

branch: https://github.com/rtsuk/coffee-script/tree/rwt-sourcemap

plan: https://github.com/rtsuk/coffee-script/wiki/Source-Map-Plan

progress: https://gist.github.com/1508440

I think I've got the hard parts figured out and should have something to ring in the new year. Advice on approach and style are very welcome.

Answer 84 · 2011-12-22T06:11:32.000Z

Here is the o() function in grammar.coffee in the version from @rtsuk:

o = (patternString, action, options) ->
  patternString = patternString.replace /\s{2,}/g, ' '
  return [patternString, '$$ = $1;', options] unless action
  action = if match = unwrap.exec action then match[1] else "(#{action}())"
  action = action.replace /\bnew /g, '$&yy.'
  action = action.replace /\$L\(([\w_$]+)\)/g, 'new yy.Location(@$1)'
  action = action.replace /\b(?:Block\.wrap|extend)\b/g, 'yy.$&'
  [patternString, "$$ = #{action};", options]

Once line number support is thrown in, you see this idiom over and over again further down in the file:

-> new Literal($1).setLocation($L(1))

This could probably be DRY'ed up by allowing o() to distinguish strings from functions.

For complicated constructions, you'd still use the functions:

new Value($1, [].concat $2).setLocation($L(1))

Then o() does its current magic.

But for simple constructs you could simplify the DSL:

"Literal $1"

If o() detects a string, it could build the equivalent of whatever '-> new Literal($1).setLocation($L(1))' expands to.

I hope this makes sense. Long story short, it would be nice to DRY up the DSL for the simple cases. On the other hand, the setLocation calls don't really offend me that much, as I think the grammar is relatively stable, and the patterns are pretty clear, even if the code is a bit ugly on the surface.

Answer 85 · 2011-12-24T14:41:26.000Z

I know people are working on their own solutions and something like sourceMap + advanced debugging in the browser would be awesome.

However, the least one needs is the line number in the compiled JS code.

I wrote a quick hack for it. available on my master branch
https://github.com/hbt/coffee-script

Run it using
coffee -cd file.coffee
-d or --debug will display the line number in a comment above the instruction in the compiled JS code

It's not much but it helps with the debugging.
Hopefully it will not take another year before this issue is fixed

Thanks

Answer 86 · 2012-01-04T09:53:24.000Z

@pmuellr In Chrome, SourceMaps just landed on Canary: http://peter.sh/2012/01/css-selector-profiler-source-mapping-and-software-rendering/

Answer 87 · 2012-01-05T21:02:24.000Z

Recording source line numbers is now done over on https://github.com/rtsuk/coffee-script, if run coffee with --nodes you can see where it thinks the nodes are in the source. If anyone sees any errors in the locations reported, please let me know.

I'm moving on to trying to track the lines in the generated source and I'm failing to see any equivalently easy way to go about it.

Right now the the base class for all the nodes has a compile method that returns the generated source as a string. Each node's compile method has no way to know where in the generated source its contribution is going to land, and it also has no way to return the source line numbers to the caller who might.

One approach might be to stop using strings for returning code but create some CodeChunk object that can retain the source mapping. These code chunks could be then assembled into the resulting JavaScript and a map built at the same time.

Can anyone think of another approach? Any comments on this one? I'm afraid it's going to be a big change no matter how we slice it, so I'd love some feedback from the core contributors.

Answer 88 · 2012-01-05T22:27:44.000Z

@rtsuk regarding "CodeChunks", I created basically that exact thing and it is in the mozilla/source-map library because anyone who wants to use source maps in their compiler is going to run in to these problems.

https://github.com/mozilla/source-map/blob/master/lib/source-map/source-node.js

There is also some documentation here (find the "Source Node" section): https://github.com/mozilla/source-map/blob/master/README.md

Hope this helps! If you find it lacking, pull requests are welcome :)

Answer 89 · 2012-01-05T22:29:08.000Z

Oh, and I guess I forgot to mention it here, but the mozilla/source-map lib is BSD licensed now. I know there was concern about the license before, but it should be a non-issue now.

Answer 90 · 2012-01-06T20:06:07.000Z

@lucaswoj @pmuellr
Actually implemented that idea.
Matching the compiled coffeescript (JS) to the original coffeescript file line by line.

How you can test it:

clone my repo

git clone git://github.com/hbt/coffee-script.git

checkout the branch

git checkout --track -b lines origin/line-numbers1-1

run cake build twice

./bin/cake build; ./bin/cake build

compile one of your coffeescript files
./coffee-script/bin/coffee -c test.coffee

produces the regular test.js

compile in debug mode
./coffee-script/bin/coffee -cd test.coffee
produces a version matching line numbers

Examples with the underscore library:
Original coffeescript file
https://gist.github.com/1572009

Compiled version
https://gist.github.com/1572005

Debug Version
https://gist.github.com/1571999
(notice the line numbers are a very close match)

Note: for whatever reason, the gist https://gist.github.com/1572009 starts at line 6 instead of line 1. Download the files and compare them in your editor as proof

Answer 91 · 2012-01-06T20:09:40.000Z

This is brilliant! I'll play with this some more when I get off work.

Lucas Wojciechowksi
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

On Friday, January 6, 2012 at 12:06 PM, hbt wrote:

@lucaswoj @pmuellr
Actually implemented that idea.
Matching the compiled coffeescript (JS) to the original coffeescript file line by line.

How you can test it:

clone my repo

git clone git://github.com/hbt/coffee-script.git (http://github.com/hbt/coffee-script.git)

checkout the branch

git checkout --track -b lines origin/line-numbers1-1

run cake build twice

./bin/cake build; ./bin/cake build

compile one of your coffeescript files
./coffee-script/bin/coffee -c test.coffee

produces the regular test.js

compile in debug mode
./coffee-script/bin/coffee -cd test.coffee
produces a version matching line numbers

Examples with the underscore library:
Original coffeescript file
https://gist.github.com/1572009

Compiled version
https://gist.github.com/1572005

Debug Version
https://gist.github.com/1571999
(notice the line numbers are a very close match)

Reply to this email directly or view it on GitHub:
#558 (comment)

Answer 92 · 2012-01-06T20:15:23.000Z

@hbt I don't think that the line-by-line equivalence is the best solution going forward, but it's great to see a working prototype.

Answer 93 · 2012-01-07T19:48:56.000Z

Even though parser-aware line number support is around the corner, I still wonder how easy it would be to automatically match up CS line numbers to JS line numbers with a simple heuristic algorithm. For example, you could scan CS code, and each time you encounter a token for the first time, then look for the first occurrence of that token in the remaining JS code. This wouldn't give you an exact line-to-line match, but it would get you down to method-level granularity pretty reliably, I think.

You could avoid false matches by blacklisting keywords and/or really small words. Also, you'd always scan forward in the JS code, which I think is reliable. You'd have to ignore "var" statements in JS, so you don't get tripped up by hoisting.

Has anybody tried this kind of approach? Here's an example, done manually:

# key off "hanoi"
CS:
  hanoi = (ndisks, start_peg=1, end_peg=3) ->
    if ndisks
JS:
  var hanoi;

  hanoi = function(ndisks, start_peg, end_peg) {
    var staging_peg;
    if (start_peg == null) start_peg = 1;
    if (end_peg == null) end_peg = 3;
    if (ndisks) {
# key off "staging_peg"
CS:
      staging_peg = 1 + 2 + 3 - start_peg - end_peg
      hanoi(ndisks-1, start_peg, staging_peg)
JS:
      staging_peg = 1 + 2 + 3 - start_peg - end_peg;
      hanoi(ndisks - 1, start_peg, staging_peg);
# key off console
CS:
      console.log "Move disk #{ndisks} from peg #{start_peg} to #{end_peg}"
      hanoi(ndisks-1, staging_peg, end_peg)

  hanoi(4)
JS:
      console.log("Move disk " + ndisks + " from peg " + start_peg + " to " + end_peg);
      return hanoi(ndisks - 1, staging_peg, end_peg);
    }
  };

  hanoi(4);

Especially for server-based code, once you had a tool that could produce something like the above, it would be a relatively straightforward matter to build tools that let you find CS code sections based on JS line numbers.

Answer 94 · 2012-01-08T07:56:25.000Z

@jashkenas and others,

Check out this demo of CS/JS line number debugging that works with zero compiler support:

http://shpaml.webfactional.com/misc/underscore.htm

I've tested it on a few files, including underscore.coffee as shown in the link. It needs a little polishing in terms of packaging, but I think the basic approach could solve a lot of the problems folks have with debugging.

Answer 95 · 2012-01-08T14:53:31.000Z

Wow @showell -- really impressive!

Answer 96 · 2012-01-08T17:40:22.000Z

@showell - I like your approach - simple and effective!

Answer 97 · 2012-01-08T18:36:10.000Z

@showell

I definitely like your approach. I've started using coffeescript about a couple of weeks ago and the debugging was a major turn off. It's the whole 80/20 rule. 80% debugging compiled JS, 20% writing code with elegant syntax

Anyway, with the post-compilation formatting, I no longer have to read the JS code when debugging and I don't need any extra tools + the console statements match, I have breakpoints in the editor and the stacktrace reports the right line number.

In my opinion, the only thing missing in your solution is an auto-formatter and as soon as I start running into problems with my hack, I will fork yours and improve it.

Thanks again for working on this!

Answer 98 · 2012-01-09T17:16:47.000Z

@geraldalewis , @ozra, and @hbt: Thanks for the feedback.

@hbt I've been nibbling away at the tool this morning, and one thing that may be of interest to you is that I've started to separate concerns a bit. So, for example, there's now a module called side_by_side.coffee, which takes the equivalent of a source map and displays the CS and JS side by side. With that broken out, it should be easier for folks to enhance their own tooling.

@geraldalewis and @ozra, On the algorithm front I've actually simplified the algorithm a bit, by basically looking only at assignment statements in CS/JS. This seems to cut down on false matches significantly, while still being fairly precise. I'm mostly using underscore.js as my litmus test, and the particular style of coding in underscore.js seems to work well. I haven't tried it out on more "class"-centric code.

Obviously, the end game is here is full-on compiler support for mappings, and that's why I'm trying to keep the tooling separate from the temporary workaround of heuristic mapping.

There's probably a happy medium, where the CS transcompiler could give just a few hints to make my heuristic algorithm even more effective. Then, you wouldn't need full-blown line number support. An example would be blank lines. If CS did nothing else but create a source map between CS-blank-lines and JS-blank-lines, I think a little bit of tooling could effectively solve debugging problems for most reasonable flexible coders.

Answer 99 · 2012-01-09T20:39:38.000Z

@showell I'm glad you've made such progress on this, since I've not been about to make much headway on the generation side and new priorities at work mean I have to stop working on it for a while.

I hope someone decides to merge in #2000 as it will at least give one the source line numbers for nodes. Even if the output source maps don't arrive for a while it could be used in #2021 to provide source file and line numbers for the new errors that will be generated.

Answer 100 · 2012-01-10T23:38:26.000Z

I've done a little more work on the non-compiler solution to line mapping.

On the heuristics side, I've gotten line mapping down to the granularity of 25-line chunks, max. For easy code, it's much more fine-grained, but I err on the side of avoiding false matches. There's definitely room for improvement on line-matching, but the basic concept's proven out. I would love for folks to test this out and maybe file some issues on particularly difficult code to match.

On the UI side, there's now a web interface, where you can examine a whole directory's worth of .coffee and .js files. The front page lists all the coffee files, and if you click on the link, it makes a best attempt to find the matching .js file, then it shows the two files side by side.

The project is here:

https://github.com/showell/CoffeeScriptLineMatcher

To launch the web server, you do something like this:

 > coffee dashboard.coffee . 3000
Server running at http://localhost:3000/

The first cmd-line arg is the directory ("." in my example), and the second one is the port. So, in the browser, you'd launch:

http://localhost:3000/

The tool should work pretty much out of the box; you just need node.js and coffee.