zom-lang/zom

Parsing Error need to point to the code

Closed this issue · 4 comments

Is your feature request related to a problem? Please describe.
Add error that clearly point to the code.

Describe the solution you'd like

  • rename the actual Token enum into TokenType
  • Create a struct Token that stores a TokenType and a range, named the span.
  • The lexer will still put the token type and add a span for every token, the start and the end.

After that the token stream has the location in the code.

  • Create a trait CodeLocation that will have a function span() that return the span of the AST.
  • All AST nodes will store it's own span, (and so implement CodeLocation)
  • When an error occurs, you can use the span of the most coherent AST node and spawn an error. -> #30

The new struct shouldn't pass a Position to the new struct but instead a much lighter one without the filename and filetext parameters. But instead put those parameters in the new struct.

I'm definitely not happy with the result, I think I need to found a new implementation, it's too bulky. I've two ideas how to do it more efficiently :

  1. Create a new method (or function) to the lexer that retokenize the text and return the position of the token that as issue with. The prototype looks like that get_pos_token(&mut self, tok_idx: usize) -> Option<(Position, Position)> with the first Position be the start and the second, the end.
  2. Or a function, with a vec of tokens recreate the text an then, because it recreates the text, it knows where the token was and return a tuple of Position (like the first idea) with the text that it recreates and the filename doesn't change.
    Both function / method will only be call when there is an error.

This is done.

The "reverse lexer" is just dumb, we can instead store a span in tokens, and in the AST