/IBITTokenizer

Simple persian text-tokenizer base on tokenizers lib

Primary LanguagePython

Stargazers