glample/fastBPE

limitVocab

geert56 opened this issue · 0 comments

Why is the limitVocab function necessary? And is it correctly implemented? I see that the query string might have a kTokenDelim attached but then the lookup in vocab makes no sense. What is going on here?