Byte Level BPE Tokenizer (GPT2/RoBERTa )

Question

Byte Level BPE Tokenizer (GPT2/RoBERTa )

sinamoeini opened this issue 3 years ago · 5 comments

Hi TensorFlow team,

Is there going to be a byte level bpe tokenizer in tensorflow text?

Answer 1 · 2021-11-09T18:58:17.000Z

BPE is already supported by SentencePiece. If you have a sentence piece model, you can use it with the sentencepiece op.

Answer 2 · 2021-11-09T19:12:56.000Z

Thank you @thuang513. So sentencepiece and byte-level BPE only differ in training phase, right? So if I have a trained byte-level BPE I should be able to use sentencepiece and just give it the vocab and merges.

Answer 3 · 2021-11-09T19:19:15.000Z

I believe so. If you trained the model using SentencePiece you should be able to use it as is.

Answer 4 · 2021-11-09T19:30:27.000Z

I dug a bit more and it seems that there is a small difference in how they treat spaces. I will try to replicate hugging face RoBERTa tokenizer using tensorflow sentence piece tokenizer and update this thread

Answer 5 · 2022-04-13T09:27:05.000Z

@sinamoeini Any updates on this?