Evaluation results and protocol of 3DPW

Question

Evaluation results and protocol of 3DPW

HospitableHost opened this issue 2 years ago · 7 comments

Hello, thank you so much for open sourcing such a nice work (PyMAF, ICCV 2021).
I use your pretrained model to evaluate on the dataset 3DPW(test split), but obtain a different result from the paper.

So do you know where i am wrong?

Answer 1 · 2023-03-14T14:10:43.000Z

In addition, I used VIBE's framework for evaluation.

Answer 2 · 2023-03-15T06:55:59.000Z

Hi, is the gender models used in the evaluation? Besides, is the root position subtracted for the PVE evaluation?

Answer 3 · 2023-03-15T10:59:50.000Z

Hi, is the gender models used in the evaluation? Besides, is the root position subtracted for the PVE evaluation?

Hey, in order to avoid mistakes, I specially used all npz files in your project to replace the vibe ones. So the gender model is the same.
PVE evaluation I used is VIBE code. And expect for PVE, MPJPE PA-MPJPE are both bad.
I just use PyMAF model to replace VIBE class, and make sure that the interface is consistent.

Answer 4 · 2023-03-18T03:05:09.000Z

Hi, have you used the PyMAF code to perform the evaluation? It may help to find out the reason causing the difference.

Answer 5 · 2023-03-18T09:58:25.000Z

Hi, have you used the PyMAF code to perform the evaluation? It may help to find out the reason causing the difference.

Yes, I used the PyMAF code to perform the evaluation, and I find that the data you use is not exactly the same as that of VIBE, although they are all from the test set.

Answer 6 · 2023-03-18T10:21:59.000Z

We follow the data and code of SPIN to conduct the evaluation of PyMAF. I have not tested it on the VIBE data and code.

Answer 7 · 2023-03-24T03:31:19.000Z

It seems that this issue is related to more than one repository such as VIBE and SPIN. I temporally close this issue. Please feel free to reopen it if anyone has more related information.