/tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

Primary LanguagePythonMIT LicenseMIT

Watchers

No one’s watching this repository yet.