huggingface/tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
RustApache-2.0
Pinned issues
Issues
- 1
batch_encode_plus doesn't work correctly
#1704 opened
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
RustApache-2.0