g-i-o-r-g-i-o/minbpe
Minimal, clean, code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
PythonMIT
Stargazers
No one’s star this repository yet.
Minimal, clean, code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
PythonMIT
No one’s star this repository yet.