/Tokenizer

Fast and customizable text tokenization library with BPE and SentencePiece support

Primary LanguageC++MIT LicenseMIT

Stargazers