Logos

Create ridiculously fast Lexers.

Logos has two goals:

To make it easy to create a Lexer, so you can focus on more complex problems.
To make the generated Lexer faster than anything you'd write by hand.

To achieve those, Logos:

Combines all token definitions into a single deterministic state machine.
Optimizes branches into lookup tables or jump tables.
Prevents backtracking inside token definitions.
Unwinds loops, and batches reads to minimize bounds checking.
Does all of that heavy lifting at compile time.

Example

 use logos::Logos;

 #[derive(Logos, Debug, PartialEq)]
 #[logos(skip r"[ \t\n\f]+")] // Ignore this regex pattern between tokens
 enum Token {
     // Tokens can be literal strings, of any length.
     #[token("fast")]
     Fast,

     #[token(".")]
     Period,

     // Or regular expressions.
     #[regex("[a-zA-Z]+")]
     Text,
 }

 fn main() {
     let mut lex = Token::lexer("Create ridiculously fast Lexers.");

     assert_eq!(lex.next(), Some(Ok(Token::Text)));
     assert_eq!(lex.span(), 0..6);
     assert_eq!(lex.slice(), "Create");

     assert_eq!(lex.next(), Some(Ok(Token::Text)));
     assert_eq!(lex.span(), 7..19);
     assert_eq!(lex.slice(), "ridiculously");

     assert_eq!(lex.next(), Some(Ok(Token::Fast)));
     assert_eq!(lex.span(), 20..24);
     assert_eq!(lex.slice(), "fast");

     assert_eq!(lex.next(), Some(Ok(Token::Text)));
     assert_eq!(lex.slice(), "Lexers");
     assert_eq!(lex.span(), 25..31);

     assert_eq!(lex.next(), Some(Ok(Token::Period)));
     assert_eq!(lex.span(), 31..32);
     assert_eq!(lex.slice(), ".");

     assert_eq!(lex.next(), None);
 }

For more examples and documentation, please refer to the Logos handbook or the crate documentation.

How fast?

Ridiculously fast!

test identifiers                       ... bench:         647 ns/iter (+/- 27) = 1204 MB/s
test keywords_operators_and_punctators ... bench:       2,054 ns/iter (+/- 78) = 1037 MB/s
test strings                           ... bench:         553 ns/iter (+/- 34) = 1575 MB/s

Acknowledgements

Pedrors for the Logos logo.

Thank you

Logos is very much a labor of love. If you find it useful, consider getting me some coffee. ☕

If you'd like to contribute to Logos, then consider reading the Contributing guide.

Contributing

Logos welcome any kind of contribution: bug reports, suggestions, or new features!

Please use the issues or pull requests tabs, when appropriate.

Releasing a new version

Note

This section is only useful to Logos' maintainers.

First, make sure you are logged-in https://crates.io with: cargo login. If you don't have write access to Logos' crates, you can still perform steps 1-4, and ask a maintainer with accesses to perform step 5.

This project uses cargo-release to publish all packages with more ease. Note that, by default, every command runs in dry mode, and you need to append --execute to actually perform the action.

Here are the following steps to release a new version:

create a branch release-x.y.z from the master branch;
run and commit cargo release version --workspace <LEVEL>;
run and commit cargo release replace --workspace;
push your branch and create a pull request;

and, once your branch was merged to master, run the following:

cargo release publish --package logos-codegen
cargo release publish --package logos-derive
cargo release publish --package logos-cli
cargo release publish --package logos

And voilà!

License

This code is distributed under the terms of both the MIT license and the Apache License (Version 2.0), choose whatever works for you.

See LICENSE-APACHE and LICENSE-MIT for details.

maciejhirsz/logos