Intern strings in the Lexer
Plecra opened this issue · 3 comments
Line 13 in 3796465
This is an easy little optimization that we can implement in the lexer - A crate like https://docs.rs/lasso is probably worth using
So, we actually do this in the hoist step:
Line 59 in 3796465
Where UniqueSymbol is a wrapper around an integer, basically.
The issue is that this step occurs too late, imo. I mentioned this on discord, but it would be ideal to do this right after parsing, which also opens up richer semantics for macros and types.
By the way, in the records branch, I've started to intern Strings in the lexer with the introduction of SharedSymbol, which is a wrapper around an unsigned integer. We actually don't have to intern any strings, so to speak, as we can just use Spans to reconstruct the string we parsed.
This is fully implemented in the records branch, so I'm closing this issue. This should be merged into master by the next release. 🚀