/anthropic-tokenizer

Approximation of the Claude 3 tokenizer by inspecting generation stream

Primary LanguagePython

No issues in this repository yet.