rb-tokenizer is a flexible, rule-based tokenizer written in Rust, designed to make text tokenization customizable and extendable. It supports a wide range of applications, from simple text parsing to complex programming language lexers.
- Customizable Tokenization: Easily define your own tokenization rules with regular expressions and symbols.
- Extensible Architecture: Add new rule types to suit your specific tokenization needs.
- Performance: Optimized for speed and efficiency, handling large texts swiftly.
- Easy Integration: Designed to be integrated into larger parsing or text analysis projects.
Ensure you have Rust installed on your system. You can download Rust and cargo via rustup.
Add rb-tokenizer to your Cargo.toml:
[dependencies]
rb-tokenizer = { git = "https://github.com/maniartech/rb-tokenizer.git" }To use rb-tokenizer in your project, start by creating a Tokenizer instance and adding rules:
use rb_tokenizer::Tokenizer;
let mut tokenizer = Tokenizer::new();
tokenizer.add_regex_rule(r"^\d+", "Number", None);
tokenizer.add_regex_rule(r"^[a-zA-Z_][a-zA-Z0-9_]*", "Identifier", None);
tokenizer.add_symbol_rule("(", "Operator", Some("OpenParen"));
tokenizer.add_symbol_rule(")", "Operator", Some("CloseParen"));
tokenizer.add_symbol_rule("+", "Operator", Some("Plus"));
let tokens = tokenizer.tokenize("ADD(2 + 2)").unwrap();
println!("{:?}", tokens);
// Output:
// Ok([
// Token { token_type: "Identifier", token_sub_type: None, value: "ADD", line: 1, column: 1 },
// Token { token_type: "Operator", token_sub_type: Some("OpenParen"), value: "(", line: 1, column: 4 },
// Token { token_type: "Number", token_sub_type: None, value: "2", line: 1, column: 5 },
// Token { token_type: "Operator", token_sub_type: Some("Plus"), value: "+", line: 1, column: 7 },
// Token { token_type: "Number", token_sub_type: None, value: "2", line: 1, column: 12 },
// Token { token_type: "Operator", token_sub_type: Some("CloseParen"), value: ")", line: 1, column: 13 }
// ])You can find more examples in the tests/ directory of the repository, demonstrating various use cases and configurations.
Contributions to rb-tokenizer are welcome! Here are a few ways you can help:
- Reporting Issues: Found a bug or have a feature request? Please open an issue.
- Pull Requests: Want to contribute code? Pull requests are warmly welcomed. Please ensure your code adheres to the project's coding standards and includes tests, if applicable.
- Documentation: Improvements to documentation or new examples are always appreciated.
Before contributing, please read our CONTRIBUTING.md guide.
rb-tokenizer is distributed under the MIT License. See LICENSE for more information.
- Inspired by the flexibility of rule-based tokenization in various programming languages and frameworks.