An Industry Standard Tokenizer, purposed for large-scale language models like OpenAI's GPT Series.
Primary LanguagePython