Decoding not working?

Question

Decoding not working?

NightShadow777 opened this issue 4 years ago · 5 comments

Hello! I decided to try to train the model for the Russian language myself and during the training or starting the generation of the text, all the texts are in the form as in the picture. Doesn't decoding happen? I looked at your code like everything is the same.
Thank you in advance for your help!

Привет! Решил попробовать сам обучить модель для русского языка и вовремя обучения или запуске генерации текста все тексты идут в таком виде как на картинке. Получается не происходит декодирование? Смотрел твой код вроде все так же.
Заранее благодарю за помощь!

Answer 1 · 2020-08-21T10:47:56.000Z

Странно. А ты точно используешь sentencepiece токенайзер (а не тот BPE, который шел с gpt-2)?

import encoder_sp as encoder

Answer 2 · 2020-08-21T11:06:00.000Z

Да использую sentencepiece и encoder_sp

Answer 3 · 2020-08-21T14:31:51.000Z

Если будет у тебя время, посмотри пожалуйста код свежим глазом. Может, где-то ошибка закралась... Мой email: rudolf_miller@ukr.net, скину доступ к серверу.

Answer 4 · 2020-08-21T19:36:08.000Z

Заменил свои sp.model и sp.vocab на твои.

Answer 5 · 2020-08-21T20:26:41.000Z

Написал, посмотри pls в почту.

Hello! I decided to try to train the model for the Russian language myself and during the training or starting the generation of the text, all the texts are in the form as in the picture. Doesn't decoding happen? I looked at your code like everything is the same. Thank you in advance for your help!

Hello! I decided to try to train the model for the Russian language myself and during the training or starting the generation of the text, all the texts are in the form as in the picture. Doesn't decoding happen? I looked at your code like everything is the same.
Thank you in advance for your help!