Error when generating with long prefix
anishthite opened this issue · 4 comments
Hi! When I generate text with a prefix longer than 4 characters, I get the following error:
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[0,0] = 1024 is not in [0, 1024)[[{{node sample_sequence_6/while/model/GatherV2_1}}]]
It does not occur if the prefix is "Hi!", but it does occur when it is "Hi, this is a longer piece of text"
Do you know why this may be happening?
The issue does not occur when using the original generate script from OpenAI
It seems to be occurring on this line:
out = sess.run(output, feed_dict={ context: batch_size * [context_tokens] })
the OpenAI script has
for _ in range(nsamples // batch_size): out = sess.run(output, feed_dict={ context: [context_tokens for _ in range(batch_size)] })[:, len(context_tokens):]
Could this difference be causing the issue?
Looking into this now since it is affecting the cloud API. It's possible this difference could affect it.
Able to reproduce, but only if the generated length is long as well (which is the default, and explains why I haven't seen the issue as I work with smaller texts)
It's possible the prefix length in tokens + generated text in tokens > 1024. It might be a good idea to cap the length at that difference.