General tokenizer library for the Web and Node. Supports Huggingface and Tiktoken formats
Primary LanguageRustBSD 2-Clause "Simplified" LicenseBSD-2-Clause