/picky_bpe

BPE modification that implements removing of the intermediate tokens during tokenizer training.

Primary LanguagePython

Stargazers