Time taken on 8 A100?

[jzhang38]

Hi,

Great work! I find this eval suite very handy. Just curious, how long would it take to perform the full evaluation for a 7B model with 8A100?

[hsiehjackson]

Hi @jzhang38, for mistralai/Mistral-7B-Instruct-v0.2 with vLLM, here is the summary of costing time per task.

4k: 4 mins
8k: 5 mins
16k: 10 mins
32k: 17 mins
64k: 40 mins
128k: 2 hrs

[jzhang38]

Thanks. That is very helpfu!