/happytoken

Ready-made tokenizer library for working with GPT and tiktoken

Primary LanguageRustMIT LicenseMIT

Stargazers