Parsing Error need to point to the code
Closed this issue · 4 comments
Larsouille25 commented
Is your feature request related to a problem? Please describe.
Add error that clearly point to the code.
Describe the solution you'd like
- rename the actual
Token
enum intoTokenType
- Create a struct
Token
that stores a TokenType and a range, named the span. - The lexer will still put the token type and add a span for every token, the start and the end.
After that the token stream has the location in the code.
- Create a trait
CodeLocation
that will have a functionspan()
that return the span of the AST. - All AST nodes will store it's own span, (and so implement
CodeLocation
) -
When an error occurs, you can use the span of the most coherent AST node and spawn an error.-> #30
Larsouille25 commented
The new struct shouldn't pass a Position to the new struct but instead a much lighter one without the filename and filetext parameters. But instead put those parameters in the new struct.
Larsouille25 commented
I'm definitely not happy with the result, I think I need to found a new implementation, it's too bulky. I've two ideas how to do it more efficiently :
- Create a new method (or function) to the lexer that retokenize the text and return the position of the token that as issue with. The prototype looks like that
get_pos_token(&mut self, tok_idx: usize) -> Option<(Position, Position)>
with the first Position be the start and the second, the end. - Or a function, with a vec of tokens recreate the text an then, because it recreates the text, it knows where the token was and return a tuple of Position (like the first idea) with the text that it recreates and the filename doesn't change.
Both function / method will only be call when there is an error.
Larsouille25 commented
This is done.
Larsouille25 commented
The "reverse lexer" is just dumb, we can instead store a span in tokens, and in the AST