.NET and Typescript implementation of BPE tokenizer for OpenAI LLMs.
Primary LanguageC#MIT LicenseMIT