About Evaluation
Closed this issue · 2 comments
52PengUin commented
Hi dear author, Thanks for the amazing work.
I tried to test the model so I followed the instructions in README
- download the pre-trained models
- run sample.py to sample the motions with the pre-trained kit-mmm-xyz model
- run evaluate.py to calculate the metrics with motions above
However, I found that the results were not good enough. I am new to this area, and don't know where I did wrong. I will be deeply graceful if you could help me find out the mistakes. Thanks
| APE_root | APE_traj | APE_mean_pose | APE_mean_joints | AVE_root | AVE_traj | AVE_mean_pose | AVE_mean_joints | |
|---|---|---|---|---|---|---|---|---|
| Paper | 0.963 | 0.955 | 0.104 | 0.976 | 0.445 | 0.445 | 0.005 | 0.448 |
| My run | 1.033 | 1.024 | 0.104 | 1.048 | 0.448 | 0.448 | 0.005 | 0.451 |
Mathux commented
Hello @52PengUin,
Actually you did not do anything wrong, the model is not exactly the same as when the one I used to produce the paper. I tried to clean up the code, and fix some seeds to be more reproducible but this made the model not exactly the same.
The difference is not very great in fact, and may also depend on the randomness of the generation.
52PengUin commented
OK, I understand the problem now. Thank you for the reply.