/rust-tokenizers

Rust-tokenizer is a drop-in replacement for the tokenization methods from the Transformers library

Primary LanguageRustApache License 2.0Apache-2.0

Watchers