Do summarization using ONNX and Rust
This example uses GPT-2. Install Python and use the requirement.txt to install everything you need.
Run the following command:
python -m transformers.onnx --model=gpt2 out/
Then run python generate.py to create the vocab and merges files that are also needed.
The current implementation doesn't generate any problems with the compiler, but only provides a single word out. According to ChatGPT this is because "The current approach takes the index with the maximum value from the output tensor, which only gives you a single word as output. Instead, you should generate the output sequence by sampling from the probability distribution of the model output.".
Please file a pull request with a plausible fix if you have one!