A lightweight byte-pair encoding tokenizer written in Rust.
Primary LanguageRustMIT LicenseMIT
This repository is not active