BPE Tokenzier geting stuck while encoding `pavement`

Question

BPE Tokenzier geting stuck while encoding `pavement`

Closed this issue 9 months ago · 2 comments

Hello,

Thank you you creating this CLIP demo app.

While playing around with the app, we noticed that the text tokenzier isn't working for the following text "cat on the pavement". It looks like the bpe function is getting suck in infinite loop while trying to encoding the token 'pavement'.

Answer 1 · 2023-10-03T05:52:03.000Z

I think I might know what's wrong. Will fix it soon :)

Answer 2 · 2023-10-03T06:05:13.000Z

Should fix it now. Please pull and check