Version 1.2.1 of @lezer/generator tokenizes differently
viktor-berglund-mw opened this issue · 2 comments
viktor-berglund-mw commented
I have a grammar like this:
@top Program { expression }
@skip { Blank }
expression { (Number | Identifier)+ }
@tokens {
Number { $[0-9]+ }
Identifier { $[0-9a-z]+ }
Blank { ' ' }
@precedence { Number, Identifier }
}
In version 1.2.0. the string 1a
is tokenizes as: Program(Identifier)
.
In version 1.2.1, 1a
is tokenized as: Program(Number,Identifier)
marijnh commented
Yes, there was a bug in versions 1.2.0 and below, causing @precedence
to not be respected in cases like this. The current behavior is the intended behavior.
viktor-berglund-mw commented
Aha thanks for the response!