jgm/djot

Why aren't comments in the AST?

alxlg opened this issue · 2 comments

alxlg commented

Why comments like {% this %} are completely skipped instead of being in the AST (and ignored when converting to, for example, HTML)?

I mean, what if one wants to use a parser to develop a Djot GUI editor? How would one enable the user to see and edit comments if they aren't in the AST at all?

jgm commented

For an editor you'd want a concrete rather than an abstract syntax tree.
Even ignoring comments, there is a many -> one mapping from concrete syntax to the AST.

  • 👍1
matklad commented

To give a bit more context here, while it is possible, with some extra engineering, to re-use the same parser for CST for an IDE and AST for "real" processing, this usually results in some combination of:

  • AST parser being slow
  • CST parser not being flexible enough (esp. around error handling, or incremental reparsing)
  • Parsing code becoming completely unreadable due to extra abstraction

I think "have two parsers" is a reasonable starting point here.