Simple Byte pair Encoding mechanism used for tokenization process . written purely in C
Primary LanguageCMIT LicenseMIT
No issues in this repository yet.