javirandor/anthropic-tokenizer
Approximation of the Claude 3 tokenizer by inspecting generation stream
PythonMIT
Stargazers
- wudangtausgrg
- dubytang
- ivawzhMelbourne
- gurusuraChennai, India
- mmathysZurich, Switzerland
- Amanoki
- abertschiZurich, Switzerland
- jw782cn
- anupampatil44Utopia (I wish)
- nigivaParis
- apiwhichway
- SkyrookieYuChangsha City, Province of Hunan, People's Republic of China
- cemilakkoc
- hamannju
- monoxgasUtah, United States
- byt3bl33d3rError: Unable to resolve
- junuMoonSeoul, South Korea
- toshVienna, Europe
- jmpazNYC
- neueccTokyo, Japan
- yika-aixi
- jerbear2008
- guitarrapcTokyo, Japan
- seohyunjun
- jake-mythotech
- javirandorZurich
- gutbashPhiladelphia
- itarutomy97
- FiazAbrar
- zhzLuke96上海, China
- sinaahmadiZurich, Switzerland
- aasthavarOdisha, India
- OSH212Montreal, Canada
- JakobWong
- chunyeahShanghai, China
- condelogan