schamane/rwkv_tokenizer

RWKV world tokenizer

This is a native port of the new RWKV tokenizer.

Usage

1. Install Rust.
2. Run python tokenizer_fast.py.

License

Licensed under either of

• Apache License, Version 2.0, (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.