/tiktoken-rs

Ready-made tokenizer library for working with GPT and tiktoken

Primary LanguageRustMIT LicenseMIT

Watchers