C++ implementation of tokenizers, including tiktoken.
Primary LanguageC++Apache License 2.0Apache-2.0