Systemcluster/kitoken
Fast and versatile tokenizer for language-models, supporting BPE and Unigram tokenization and usable in native and WASM environments
RustBSD-2-Clause
Fast and versatile tokenizer for language-models, supporting BPE and Unigram tokenization and usable in native and WASM environments
RustBSD-2-Clause