NUL in Lexer input

[pmehnert]

Files containing a null character will lead to the Lexer yielding a premature EndOfStream token rather than an exception for an unexpected character. This is caused by ReaderCharIterator yielding NUL after the input stream has ended.

Consider the example below. Here the Lexer terminates without exception, but does not yield the tokens after the NUL.

int x = 0; \0 int y = 1;