glample/fastBPE

Hi, awesome work! But does it support to add special tokens for tokenization?

jianliu-ml opened this issue · 0 comments

jianliu-ml commented
Hi, awesome work! But does it support to add special tokens for tokenization?