/anthropic-tokenizer

Approximation of the Claude 3 tokenizer by inspecting generation stream

Primary LanguagePythonMIT LicenseMIT

Stargazers