/rlox-tokenizer

Converts Lox source code into syntax tokens.

Primary LanguageRustApache License 2.0Apache-2.0

rlox-tokenizer

Converts Lox source code into syntax tokens.

Lox is a toy scripting language featured in Bob Nystrom's book Crafting Interpreters. This crate provides methods for the initial transformation of Lox source code into abstract syntax tokens.

This implementation differs from the C reference implementation, most notably:

  • Full support for UTF-8 source code
  • Support for nested multiline comments
  • Doesn't require a magic EOF syntax token
  • Rust-style error handling with Result

Usage

Inside Cargo.toml:

# ...
[dependencies]
rlox-tokenizer = { git = "https://github.com/ben01189998819991197253/rlox-tokenizer.git" }

Documentation

cargo test && cargo doc --no-deps --open

Contributing

Feel free to submit a pull request or file an issue! Please run your code through rustfmt first, though.