Parse without allowing ignored tokens
rnett opened this issue · 2 comments
I want to allow whitespace most of the time, so I have token("\\s+", true). However, in a few scenarios, I want to disallow whitespace between tokens/subexpressions. A relatively easy way to do this would be nice.
The best for my use would be a version of */and that doesn't ignore anything.
Normally this is implemented as so-called 'lexer modes' which the lexer switches upon encountering a token of a specific type. A popular example is switching to the comment tokenizing mode for the rest of the line when a // occurs.
I do have plans to implement tokenizing modes in better-parse when I have more time.
@rnett Would anything like that cover your use case?
Possibly? It would be overkill by quite a bit though. I'm trying to parse dice, e.g. 5d12, and want 5d12 to parse but not 5 d 12. I'm just using another token for it at the moment and it works. Another example would be not allowing spaced before the left paren in function calls, e.g. foo(1) but not foo (1).