Refactor token streaming
Closed this issue · 0 comments
SwiftStudies commented
At the moment there is little difference in implementation (and therefore potentially expected memory/performance profiles as well as behavioural characteristics) for streams, homogenous and heterogenous ASTs.
These areas could be refactored to improve
- Ease of consumption
- Provided the expected benefits (lazy and therefore low memory consumption)
In order to do this the following should be provided:
- Streaming should identify "trigger" tokens that will be forwarded, but no others. Consideration will have to be given to range management on the node stack (or a new equivalent) but nodes should not be created
- There is an attempt to leverage commonality in behaviour between homogenous and heterogenous AST generation. This should be preserved but without the complexity of having to provide an IR and a constructor and still being left in a situation where a lot of casting has to be done