Systemcluster/kitoken
Fast and versatile tokenizer for language models with BPE, Unigram and WordPiece tokenization. Compatible with SentencePiece, Tokenizers, Tiktoken and more.
RustBSD-2-Clause
No issues in this repository yet.
Fast and versatile tokenizer for language models with BPE, Unigram and WordPiece tokenization. Compatible with SentencePiece, Tokenizers, Tiktoken and more.
RustBSD-2-Clause
No issues in this repository yet.