Time taken on 8 A100?
jzhang38 opened this issue · 3 comments
jzhang38 commented
Hi,
Great work! I find this eval suite very handy. Just curious, how long would it take to perform the full evaluation for a 7B model with 8A100?
hsiehjackson commented
Hi @jzhang38, for mistralai/Mistral-7B-Instruct-v0.2
with vLLM, here is the summary of costing time per task.
4k: 4 mins
8k: 5 mins
16k: 10 mins
32k: 17 mins
64k: 40 mins
128k: 2 hrs
jzhang38 commented
Thanks. That is very helpfu!