google/sentencepiece

HELP NEEDED Mask Token in SentencePiece tokenizer HELP NEEDED

Closed this issue · 1 comments

T5 using sentencepiece-tokenizer ,I want to perform masked language modeling,
In SentencePiece only There are only BOS,EOS,UNK,PAD so no mask token ,so how to create and add a mask token when the model is called from preteset