Approximation of the Claude 3 tokenizer by inspecting generation stream
Primary LanguagePythonMIT LicenseMIT