/multi-tokenizer

Tokenization of multilingual texts using language-specific tokenizers

Primary LanguageJupyter NotebookMIT LicenseMIT

Stargazers