tiktoken is a fast BPE tokeniser for use with OpenAI's models.
Primary LanguagePythonMIT LicenseMIT
No one’s watching this repository yet.