Suspected errors in template

[jinulee-v]

Hello, thanks for the great dataset!

I suspect that several errors are present in the template (StepGame/Code/template.py).
If I understood the codes and the templates correctly, right, below, upper-right, upper-left templates should denote BB is right/.../upper-left to AA. However, some templates seem to be inversed.

Suspicious templates:

• AA is at the bottom of BB and is on the same vertical plane. (below)
• AA presents below BB. (below)
• Object A is above object BB and to the right of it, too. (upper-right)
• AA is diagonally to the upper right of BB. (upper-right)

I'm not sure if there is more, but I guess that this is probably it.

[ZhengxiangShi]

Hi Jinu,

Thank you so much for your message and for pointing it out. I have updated the template.py file.

Best regards,
Zhengxiang

[jinulee-v]

@ZhengxiangShi , thank you for your kind and instant response.

Actually, this error also affects the actual datasets. This example is from TrainVersion/qa1_tst.json:

You might notice that the label should be "below", not "above" in this case.

I manually cleansed the dataset for my experiments, by excluding these errorneous templates.
However, I believe that you should consider re-publishing the dataset after cleansing (both in Github and Huggingface!), as the authors of CLUTRR did.

Note: This ACL 2023 findings paper also has discovered the template errors, so maybe they should get some credit. They claim that ~10.7% of the test sets have errorneous labels.

[ZhengxiangShi]

Thanks a lot for pointing it out, especially for the ACL paper. You are right, and we will have a look and correct them.

[ZhengxiangShi]

Solved by the latest commit.