keep the whole token stream and link each token to the most appropriate AST object
markusrosskopf opened this issue · 0 comments
markusrosskopf commented
Motivation: Access the AST object behind a click or hover action within the vscode extention while only providing it's location. Common features like go-to def, show references or show user defined documentation need the correct AST object to work properly.
Tasks:
- add ast_link to Token class, initially None
- add set_ast_link method to AST Node class, which sets the link to self (we do it this way round to avoid having a circular dependency between token / ast nodes)
- create a new lexer class (Token_Stream) that basically just delegates to the existing lexer class, but which remember the entire token stream (see https://github.com/florianschanda/miss_hit/blob/master/miss_hit_core/m_lexer.py#L1203 for an example)
- (this is the big step) when building the AST in the parser, pedantically call set_ast_link for every token. Note that this needs to capture all tokens, including e.g. brackets
- (optional, but a good idea to find bugs) when using the Token_Stream, do a post-processing step to assert that all tokens in the stream have an assigned ast object after parsing
After this you can go from source location to the correct token by just going over the list comparing start/end lexpos, and then you can follow ast_link to find the AST object relating to this token.