/Tokenizer

Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.

Primary LanguageC#MIT LicenseMIT

Stargazers