hidet-org/hidet

[Question] How to use for model seq2seq

batman-do opened this issue · 3 comments

batman-do commented

I want to use with model seq2seq as T5,...., How to use ?

yaoyaoding commented

Hi @batman-do,

Currently, Hidet only supports static input shape. When using hidet for text-generation task, we need to repeat the compilation for different input shapes (the sequence length will change), which is prohabitively slow.

We are working on the dynamic input size support and put it as a high priority task (together with the quantized operators). Please stay tuned, and thanks for your interest in Hidet!

  • 👍1
yaoyaoding commented

Hi @batman-do,

There is also a workaround here. We can pad the input tokens to a large length (e.g., the length of input + number of tokens to generate) and use the static shape for inference. I will give a demo on GPT-2 later.

yaoyaoding commented

The example has been merged, which uses GPT-2 as an example.
See https://github.com/hidet-org/hidet/tree/main/examples/gpt-2