/tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Primary LanguageRustApache License 2.0Apache-2.0

Pinned issues

ByteLevelBPETokenizer output seems weird

#203 opened

Open2

Training a model from in-memory data

#198 opened

Closed1

Issues