tiktoken.cpp is a faster BPE tokeniser for use with OpenAI's models.
Primary LanguageC++MIT LicenseMIT
This repository is not active