MMMU-Benchmark/MMMU

Why is every answer in Structural Engineering just "?"

mckinziebrandon opened this issue · 1 comments

I was browsing the test set with the Dataset Viewer on HuggingFace (Link) and noticed that, for the Structural Engineering subset of Architecture_and_Engineering, literally every single answer and explanation is equal to "?". Surely this is a bug?

image

Oh, never mind. I guess labels aren't provided for test, so users can't evaluate their own models.