/gpt-tokenizer

JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 and Anthropic's Claude 2 and Claude Instant. Port of OpenAI's tiktoken with additional features.

Primary LanguageTypeScriptMIT LicenseMIT

No issues in this repository yet.