MMMU-Benchmark/MMMU

Answer not present in the model prediction

Closed this issue · 2 comments

insafim commented

When using the cot prompt, didn't you encounter instances where the A/B/C/D answer is absent in the model output.

So in that case using regex to extract answers will not work leading to lower accuracy.

insafim commented

@xiangyue9607 @NipElement Can you please clarify this.

Zheng0428 commented

If the model output lacks the A/B/C/D answers, it indicates poor instruction-following ability or an inability to make a clear judgment, which reasonably contributes to lower accuracy.

  • 👍1