tiktoken.el is an Emacs Lisp port of OpenAI's tiktoken library for BPE tokenization
Primary LanguageEmacs LispGNU General Public License v3.0GPL-3.0