Reproducing the results.
Closed this issue · 4 comments
I reproduce the results using provided sctips and I got below results.
It looks like the results are different that I expected from the paper.
Reported Main Paper (Table 1) [IoU75 | 5° 2 cm | 5° 5 cm | 10° 2 cm | 10° 5 cm]
SPD: 27.0 (26.5) | 19.1 (19.1) | 21.2 (21.2) | 43.5 (42.3) | 54.0 (54.0)
SPD + Ours 43.6 (37.7) | 45.8 (43.3) | 54.4 (53.8) | 61.4 (54.9) | 73.1 (73.5)
My reproduced results are in the bracket (). I observed that the IoU75 and 10° 2 cm metrics shows large gap between reported and reproduced results. I also share my logs. Is there anything I missed to get the reported result??
[1220_190029 core.catre.engine.catre_custom_evaluator@84]: load gts of nocs_test_real
100%|##########| 2754/2754 [00:00<00:00, 24369.44it/s]
2754it [00:36, 75.57it/s]
[1220_190106 core.catre.engine.catre_custom_evaluator@318]:
objects bottle bowl camera can laptop mug Avg(6)
IoU25 53.25 99.95 85.12 70.47 84.46 86.64 79.98
IoU50 49.89 99.95 43.37 69.95 82.04 62.66 67.97
IoU75 13.09 77.13 3.38 29.77 35.48 0.00 26.48
re5te2 21.59 50.54 0.00 37.90 4.56 0.01 19.10
re5te5 23.18 54.00 0.00 42.69 7.04 0.24 21.19
re10te2 69.44 75.82 0.18 80.43 24.54 3.22 42.27
re10te5 75.95 80.30 0.20 91.59 65.31 10.42 53.96
re10te10 87.05 80.30 0.20 91.59 65.90 10.42 55.91
re5 35.87 54.00 0.02 45.49 7.06 0.24 23.78
re10 91.90 80.30 0.45 91.73 65.98 10.42 56.80
te2 80.66 94.69 34.76 87.05 29.05 26.18 58.73
te5 88.22 100.00 94.65 99.92 94.37 97.89 95.84
[1220_190258 core.catre.engine.catre_custom_evaluator@221]: Eval recalls of results at iter=4...
2754it [00:37, 72.97it/s]
[1220_190336 core.catre.engine.catre_custom_evaluator@318]:
objects bottle bowl camera can laptop mug Avg(6)
IoU25 55.95 99.95 90.26 70.54 84.89 99.06 83.44
IoU50 49.51 99.95 67.78 70.33 82.76 87.53 76.31
IoU75 24.86 90.78 5.12 34.85 68.96 1.62 37.70
re5te2 49.35 90.84 0.98 64.30 52.05 2.39 43.32
re5te5 54.38 95.13 1.05 78.58 79.55 13.79 53.75
re10te2 85.47 95.19 4.45 83.19 55.60 5.49 54.90
re10te5 92.10 100.00 5.72 99.65 91.90 52.02 73.56
re10te10 99.92 100.00 5.83 99.69 93.13 52.20 75.13
re5 84.63 95.30 1.06 80.52 79.89 13.87 59.21
re10 99.92 100.00 5.93 99.69 93.13 52.20 75.14
te2 85.47 95.19 47.64 83.19 56.13 8.24 62.64
te5 92.22 100.00 92.84 99.92 94.72 99.09 96.47
You can verify this assumption by replacing your cached nocs_test dataset file with mine.
dataset_dicts_nocs_test_real_1af30c592b38063e1d8eb5325ec7ea5e.pkl.zip
@shanice-l Thanks for your quick reply. I checked quickly using your cache files but got errors during loading the depth. As you pointed out, I think I need to check the data processing again. I will try again and share the results.
AttributeError: Caught AttributeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/miruware/anaconda3/envs/catre/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "/home/miruware/anaconda3/envs/catre/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/miruware/anaconda3/envs/catre/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/miruware/ssd_4tb/workspace/CATRE/core/catre/../../core/catre/datasets/data_loader.py", line 866, in __getitem__
return self.read_data(dataset_dict)
File "/home/miruware/ssd_4tb/workspace/CATRE/core/catre/../../core/base_data_loader.py", line 95, in read_data
return self.read_data_test(dataset_dict)
File "/home/miruware/ssd_4tb/workspace/CATRE/core/catre/../../core/catre/datasets/data_loader.py", line 712, in read_data_test
depth = load_depth(depth_path) # uint16
File "/home/miruware/ssd_4tb/workspace/CATRE/core/catre/../../core/utils/cat_data_utils.py", line 450, in load_depth
if len(depth.shape) == 3:
AttributeError: 'NoneType' object has no attribute 'shape'
Thanks, @shanice-l, I got similar results reported in the paper. I also left my logs for future people.
In my case, I don't known exact the reason why it didn't work before, but when I reset all the data processes and downloaded nocs_results from here, #6 (comment), work for me.
[IoU75 | 5° 2 cm | 5° 5 cm | 10° 2 cm | 10° 5 cm]
SPD + Ours 43.6 (44.0) | 45.8 (46.1) | 54.4 (54.8) | 61.4 (61.8) | 73.1 (73.6)
My reproduced results are in the bracket ().
[1221_172940 core.catre.engine.catre_custom_evaluator@221]: Eval recalls of results at iter=4...
2754it [00:35, 77.98
it/s]
[1221_173016 core.catre.engine.catre_custom_evaluator@318]:
objects bottle bowl camera can laptop mug Avg(6)
IoU25 55.95 99.95 90.40 70.54 84.81 99.22 83.48
IoU50 49.47 99.95 67.78 70.33 82.80 90.82 76.86
IoU75 24.37 91.60 5.22 33.97 68.77 39.82 43.96
re5te2 53.01 92.25 0.37 64.86 52.27 13.92 46.11
re5te5 57.77 96.95 0.49 79.31 79.68 14.82 54.84
re10te2 86.07 95.17 3.95 82.77 55.63 47.45 61.84
re10te5 92.10 100.00 5.42 99.65 91.99 52.60 73.62
re10te10 99.92 100.00 5.55 99.69 93.11 52.62 75.15
re5 82.82 97.07 0.49 81.43 79.91 14.82 59.42
re10 99.92 100.00 5.60 99.69 93.11 52.62 75.16
te2 86.07 95.17 47.50 82.77 56.19 83.09 75.13
te5 92.22 100.00 92.78 99.92 94.98 99.67 96.59