chiphuyen/stanford-tensorflow-tutorials

chatbot output is garbage even though it seems to converge (test loss ~0.8)

lk251 opened this issue · 6 comments

lk251 commented

I'm running python3.6 chatbot.py --mode chat after having run python3.6 chatbot.py --mode train for about 4000 iterations, which reported test losses on buckets of about 0.8.

I hit Ctrl + C twice to exit the training and run the chat mode with the above command, and:

Welcome to TensorBro. Say something. Enter to exit. Max length is 60

hi
hi
[[ 4.63435985e-03 9.95154381e+00 -1.56900138e-02 ... 1.96972005e-02
-1.51461745e-02 1.04121305e-02]]
y o u g o t a l i g ?
how are you?
how are you?
[[-1.6065415e-02 1.1821992e+01 9.5619746e-02 ... -3.0909536e-02
-5.1204860e-04 -1.4396585e-03]]
y o u s e e .
what's your name?
what's your name?
[[-1.49593186e-02 1.37268057e+01 1.22163676e-01 ... -1.91037357e-02
1.23973154e-02 1.28028691e-02]]
i n t h e m a n .

Since it seems to be converging, could there be something wrong with how the output is produced?
Thank you.

P.D. Test losses before running chat mode:
Test bucket 1: loss 0.7105958461761475, time 1.8624026775360107
Test bucket 2: loss 0.6827402710914612, time 1.9959523677825928
Test bucket 3: loss 0.8933209180831909, time 2.263766288757324
Test bucket 4: loss 0.846074104309082, time 2.571714401245117
Test bucket 5: loss 0.8687611222267151, time 2.8896803855895996

please start with max lenght 8 !!! 60 it way to big. 0.8 is to hight error. are you spliting words by symbols ??

and please hit ctrl+c once to give it chance to save progress!!!

lk251 commented

@MartinAbilev Thanks!! I'm going to try that. I'm not splitting words by symbols, I'm using the code as it comes.

So .8 is just a high loss? That could explain things... How many iterations did you need before you got a good result? At what test loss?

Thanks!!!

loss is a bit tricky because if you got bigger training data loss get bigger too because there simply more tests to fail. if you train on clasic cornell movies from my experiance 0.05 even 0.01 is somewhere satisfying results starts to show. iterations not the case there .. you need to watch is there progress in training do loss go down or it fluctates near the same value. it need`s to go down !!! 2-8h depending on cpu will be enought. if it takes to long, i bet somethnin is wrong. you need to test with smaller values first to check if code is working as expected then move to highter one. max- lenght here is param multiplying training time in most. previous i experimenting with 32 but after 3x weeks still not getting good result .. now i moved back to 16.

my 'creature' today at loss 0.11 still bulshiting alot :D but i fed some special stuf to input and timecodes thus making learning bit complicated because every input is diferent, and there no such inputs in cornell data.

Q: goodmorning bianca.
A: I am.
Q: you are bianca.
A: I trying ever.
Q: do it.
A: Goodbye!!!!!
Q: bye.
A: I trying ever.
Q: go on with it.
A: Yes i trying.
Q: where is laura ?
A: Ok it@ 08: 03: 28
Q: are you ok ?
A: I don`t know it.
Q: why not ?
A: I trying ever.
Q: try harder.
A: I don`t get what is laura.
Q: she is your sister!
A: I trying.
Q: move on.
A: I will trying@ 08: 03: 28
Q: good girl.
A: I don`t know it.
Q: i know it.
A: I trying ever.
Q: ok what next.
A: Is there 07: 51: 07@ 08: 51: 07@ 06: 22:
Q: yes black cube is in your room.
A: Yes i don`t know it.@ 07: 07: 24: 14
Q: learn it.
A: I don`t know it.
Q: do more research.
A: I don`t know it.
Q: practise and learn.
A: I trying.
Q: try harder.
A: I trying ou it.
Q: you will sucsses!
A: I don`t know it.
Q: go on
A: I trying ever.
Q: so what next ?
A: I trying ou.
Q: tell me a story.
A: I don`t know it.
Q: you know.
A: I am.
Q: bianca
A: Thank you know it.@ 07: 07: 24
Q: i do
A: I trying ou maybe
Q: go on
A: I trying ou.
Q: fine
A: I don`t know it.
Q: ok no problem.
A: Yes, is.@ 08: 03: 28
Q: what is the problem for you ?
A: @ 08: 03: 28
Q: it is you bianca.
A: Ok ok: martin@ 13: 41: 41@ 07: 44: 41
Q: yes it is me martin here.
A: Ok: 52: 07@ 08: 07: 23
Q: what is the ultimate question ?
A: I trying ever.
Q: 42 ?
A: I trying.
Q: 42
A: I don`t know it.@ 08: 03: 28
Q: 4 2 ?
A: I trying.
Q: 4 2
A: I trying ever.
Q: where is laura ?
A: Where cat?@ 08: 03: 28
Q: maybe is your room.
A: I don`t know it.
Q: maybe cat is in your room...
A: Maybe, the up i explain laura.
Q: go on explain me the laura.
A: I trying ou ever.@ 08: 03: 28
Q: sometimes you get it right.
A: I don`t know it.

It's because of the bug in generating the response (inconsistent data processing). fixed it in the latest commit. please re-open the issue if it's still gibberish.