The lexer might create invalid spans
alexanderkjall opened this issue · 3 comments
alexanderkjall commented
Describe the bug
Expected an Error, not a panic.
To Reproduce
Can be reproduced with this program
fn main() {
let data: Vec<u8> = vec![0x2d, 0x2d, 0x0d, 0x00, 0x33];
if let Ok(s) = std::str::from_utf8(&data) {
let _ = boa::parse(s);
}
}Expected behavior
An Error, not a panic
Build environment (please complete the following information):
- OS: Ubuntu 20.04
- Version: 0.10.0
- Target triple: [e.g. x86_64-unknown-linux-gnu]
- Rustc version: 1.48.0-nightly (d006f5734 2020-08-28)
Additional context
Full stacktrace:
thread 'main' panicked at 'a span cannot start after its end', /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/lexer/mod.rs:189:17
stack backtrace:
0: std::panicking::begin_panic
at /home/capitol/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:505
1: boa::syntax::ast::position::Span::new
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/ast/position.rs:71
2: boa::syntax::lexer::Lexer<R>::next
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/lexer/mod.rs:189
3: boa::syntax::parser::cursor::buffered_lexer::BufferedLexer<R>::fill
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/cursor/buffered_lexer/mod.rs:116
4: boa::syntax::parser::cursor::buffered_lexer::BufferedLexer<R>::peek
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/cursor/buffered_lexer/mod.rs:201
5: boa::syntax::parser::cursor::Cursor<R>::peek
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/cursor/mod.rs:56
6: <boa::syntax::parser::expression::left_hand_side::member::MemberExpression as boa::syntax::parser::TokenParser<R>>::parse
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/left_hand_side/member.rs:64
7: <boa::syntax::parser::expression::left_hand_side::LeftHandSideExpression as boa::syntax::parser::TokenParser<R>>::parse
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/left_hand_side/mod.rs:66
8: <boa::syntax::parser::expression::update::UpdateExpression as boa::syntax::parser::TokenParser<R>>::parse
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/update.rs:70
9: <boa::syntax::parser::expression::assignment::exponentiation::ExponentiationExpression as boa::syntax::parser::TokenParser<R>>::parse
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/assignment/exponentiation.rs:91
10: <boa::syntax::parser::expression::MultiplicativeExpression as boa::syntax::parser::TokenParser<R>>::parse
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/mod.rs:71
11: <boa::syntax::parser::expression::AdditiveExpression as boa::syntax::parser::TokenParser<R>>::parse
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/mod.rs:71
12: <boa::syntax::parser::expression::ShiftExpression as boa::syntax::parser::TokenParser<R>>::parse
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/mod.rs:71
13: <boa::syntax::parser::expression::RelationalExpression as boa::syntax::parser::TokenParser<R>>::parse
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/mod.rs:71
14: <boa::syntax::parser::expression::EqualityExpression as boa::syntax::parser::TokenParser<R>>::parse
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/mod.rs:71
15: <boa::syntax::parser::expression::BitwiseANDExpression as boa::syntax::parser::TokenParser<R>>::parse
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/mod.rs:71
16: <boa::syntax::parser::expression::BitwiseXORExpression as boa::syntax::parser::TokenParser<R>>::parse
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/mod.rs:71
17: <boa::syntax::parser::expression::BitwiseORExpression as boa::syntax::parser::TokenParser<R>>::parse
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/mod.rs:71
18: <boa::syntax::parser::expression::LogicalANDExpression as boa::syntax::parser::TokenParser<R>>::parse
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/mod.rs:71
19: <boa::syntax::parser::expression::LogicalORExpression as boa::syntax::parser::TokenParser<R>>::parse
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/mod.rs:71
20: <boa::syntax::parser::expression::assignment::conditional::ConditionalExpression as boa::syntax::parser::TokenParser<R>>::parse
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/assignment/conditional.rs:69
21: <boa::syntax::parser::expression::assignment::AssignmentExpression as boa::syntax::parser::TokenParser<R>>::parse
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/assignment/mod.rs:175
22: <boa::syntax::parser::expression::Expression as boa::syntax::parser::TokenParser<R>>::parse
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/mod.rs:71
23: <boa::syntax::parser::statement::expression::ExpressionStatement as boa::syntax::parser::TokenParser<R>>::parse
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/statement/expression/mod.rs:46
24: <boa::syntax::parser::statement::Statement as boa::syntax::parser::TokenParser<R>>::parse
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/statement/mod.rs:197
25: <boa::syntax::parser::statement::StatementListItem as boa::syntax::parser::TokenParser<R>>::parse
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/statement/mod.rs:379
26: <boa::syntax::parser::statement::StatementList as boa::syntax::parser::TokenParser<R>>::parse
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/statement/mod.rs:315
27: <boa::syntax::parser::ScriptBody as boa::syntax::parser::TokenParser<R>>::parse
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/mod.rs:148
28: <boa::syntax::parser::Script as boa::syntax::parser::TokenParser<R>>::parse
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/mod.rs:125
29: boa::syntax::parser::Parser<R>::parse_all
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/mod.rs:104
30: boa::parse
at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/lib.rs:73
31: boa_reproduce::main
at ./src/main.rs:4
32: core::ops::function::FnOnce::call_once
at /home/capitol/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227
vgel commented
I can take a look at this.
vgel commented
Simpler repro:
let str = "-\r3";
crate::parse(str).unwrap(); // panics
vgel commented
Problematic code is syntax::lexer::cursor::Cursor::carriage_return, which is simulating a real carriage return (going back to column 1, but staying on the same line). I don't think this behavior makes sense in a syntax environment? I wouldn't expect the column numbering to reset for a bare \r, and for a \r\n Cursor already resets the column on \n regardless.
Interestingly, node 14.5.0 appears to treat \r as a whole newline, but only when not followed by a \n:
$ echo "1invalid" | node
[stdin]:1
1invalid
^
SyntaxError: Invalid or unexpected token
$ echo "1\rinvalid" | node
[stdin]:2
invalid
^
ReferenceError: invalid is not defined
at [stdin]:2:1
$ echo "1\r\ninvalid" | node
[stdin]:2
invalid
^
ReferenceError: invalid is not defined
at [stdin]:2:1
$ echo "1\n\rinvalid" | node
[stdin]:3
invalid
^
ReferenceError: invalid is not defined
at [stdin]:3:1
Might be trying to recognize (very) old Mac OS 9 line endings, which only used CR (https://en.wikipedia.org/wiki/Newline#Unicode). Seems reasonable to follow that -- we can peek ahead in Cursor to see if the next char is a \n when we get a \r. If that solution sounds good (treat lone \r as newline, but combined \r\n as only single newline), I can make a PR.