Cannot reproduce the results in the paper

Question

Cannot reproduce the results in the paper

Closed this issue 2 years ago · 7 comments

Hi, thank you for open source this work, I retrained your model without changing the code, but the experimental results are quite different (some metrics differ by nearly 10 percent), I would like to know what caused this, the following are my environment settings:
Pytorch 1.7.0
Cuda 11.0.3
A single RTX3090

Answer 1 · 2022-06-19T04:52:37.000Z

Hi Liu. Have you solved the performance gap issue?

Answer 2 · 2022-06-29T03:13:22.000Z

Hi Liu. Have you solved the performance gap issue?

Hi, I am sorry for the late reply. I got a drop in accuracy because of using 'camera_composed_depth' as training data.

Answer 3 · 2023-03-04T08:37:56.000Z

Hi Liu. Have you solved the performance gap issue?

Hi, I am sorry for the late reply. I got a drop in accuracy because of using 'camera_composed_depth' as training data.
Hi Liu.I also got the performence gap without changing the code.Did you finally get the accuracy shown in paper ? I didn't find anywhere about 'camera_composed_depth' in the code,so I don't know how to solve this problem.Can you give me some advice?

Answer 4 · 2023-03-04T13:03:36.000Z

Hi, I have got a similar accuracy shown in the paper. I used to use the depth data with background in the NOCS-CAMERA dataset for training, which led to a decrease in accuracy.

Answer 5 · 2023-03-06T06:20:19.000Z

Tks for your nice reply! I'm sorry to bother again. I change the code in pose_dataset.py like below to use composed_full_depth in training according to your advice, but the accuracy didn't go up. Do I need more code modification? I'll very appreciate recieving your advice!

Answer 6 · 2023-03-06T08:34:19.000Z

Tks for your nice reply! I'm sorry to bother again. I change the code in pose_dataset.py like below to use composed_full_depth in training according to your advice, but the accuracy didn't go up. Do I need more code modification? I'll very appreciate recieving your advice!

Hi, you may have misunderstood what I mean, the author used the depth map without background depth before. I lost accuracy due to using ‘composed_full_depth’.

Answer 7 · 2023-03-12T05:31:44.000Z

Tks for your nice reply! I'm sorry to bother again. I change the code in pose_dataset.py like below to use composed_full_depth in training according to your advice, but the accuracy didn't go up. Do I need more code modification? I'll very appreciate recieving your advice!

Hi, you should be able to solve this problem by running 'bash eval.sh', because the author did not use mask-rcnn for segmentation in the test program during training.