I have a question for the file pykp/io.py
Closed this issue · 2 comments
houwenxin commented
When reading your code, I found code as follows:
if opt.src_seq_length_trunc and len(src) > opt.src_seq_length_trunc: src_tokens = src_tokens[:opt.src_seq_length_trunc]
I am puzzled about why it is len(src) here. I think we are calculating the number of words in the source sequence. Maybe it should be len(src_tokens) ?
houwenxin commented
The code is at around line 287 in io.py.
memray commented
Yes, you are right. The current code is truncating src by the string length. Fixed now.