Michael-F-Bryan/scad-rs

Preserve whitespace while parsing

Michael-F-Bryan opened this issue · 0 comments

At the moment, we deliberately filter out whitespace and comment tokens when constructing the Input for our Parser.

impl<'a> Input<'a> {
fn new(tokens: impl IntoIterator<Item = (SyntaxKind, &'a str)>) -> Self {
// FIXME: Blindly skipping whitespace and comments like this means
// Rowan will never see them, so all our token spans will be wrong.
let tokens = tokens
.into_iter()
.filter(|(k, _)| *k != SyntaxKind::WHITESPACE && *k != SyntaxKind::COMMENT)
.collect();
Input(tokens)
}

Ideally, we would have a way to retain these tokens while making sure the parser never sees them, but a naive linear scan in Input::at() would have a massive impact on performance.