About the eval result
Opened this issue · 8 comments
Hi @djiajunustc .Thank you for your great work!
I have trained your project about 120epoch. But the result was not so good as our article showed. I don't know what's the problem?
Car AP@0.70, 0.70, 0.70:
bbox AP:97.5955, 88.4117, 88.2141
bev AP:89.6285, 86.6193, 85.4555
3d AP:87.7399, 76.9524, 76.1327
aos AP:97.39, 88.01, 87.65
Car AP_R40@0.70, 0.70, 0.70:
bbox AP:98.3861, 91.5014, 91.3330
bev AP:94.9682, 87.8349, 85.8327
3d AP:90.7695, 78.6636, 76.6187
aos AP:98.19, 91.05, 90.69
Car AP@0.70, 0.50, 0.50:
bbox AP:97.5955, 88.4117, 88.2141
bev AP:97.7538, 88.7638, 88.6693
3d AP:97.6798, 88.6881, 88.5662
aos AP:97.39, 88.01, 87.65
Car AP_R40@0.70, 0.50, 0.50:
bbox AP:98.3861, 91.5014, 91.3330
bev AP:98.5041, 93.9677, 93.9661
3d AP:98.4636, 93.8064, 93.6006
aos AP:98.19, 91.05, 90.69
Pedestrian AP@0.50, 0.50, 0.50:
bbox AP:67.7471, 61.2331, 59.2942
bev AP:61.3480, 55.2211, 51.6408
3d AP:58.0767, 51.3151, 48.2123
aos AP:63.95, 56.94, 54.73
Pedestrian AP_R40@0.50, 0.50, 0.50:
bbox AP:67.9662, 61.7473, 58.9460
bev AP:61.5746, 54.5980, 50.8662
3d AP:57.2521, 50.3834, 46.4734
aos AP:63.70, 56.80, 53.80
Pedestrian AP@0.50, 0.25, 0.25:
bbox AP:67.7471, 61.2331, 59.2942
bev AP:70.3663, 65.6778, 63.1834
3d AP:70.3604, 65.6465, 63.0975
aos AP:63.95, 56.94, 54.73
Pedestrian AP_R40@0.50, 0.25, 0.25:
bbox AP:67.9662, 61.7473, 58.9460
bev AP:71.4780, 65.7675, 62.9850
3d AP:71.4710, 65.7332, 62.9239
aos AP:63.70, 56.80, 53.80
Cyclist AP@0.50, 0.50, 0.50:
bbox AP:91.2968, 75.3031, 73.5625
bev AP:84.3340, 70.1275, 67.9271
3d AP:82.7952, 67.5019, 63.0085
aos AP:89.38, 71.15, 69.25
Cyclist AP_R40@0.50, 0.50, 0.50:
bbox AP:92.9363, 77.8973, 74.7406
bev AP:88.1583, 70.3173, 66.9044
3d AP:86.1416, 67.1941, 63.7919
aos AP:90.91, 73.02, 69.94
Cyclist AP@0.50, 0.25, 0.25:
bbox AP:91.2968, 75.3031, 73.5625
bev AP:89.5904, 72.3610, 70.6035
3d AP:89.5904, 72.3610, 70.6035
aos AP:89.38, 71.15, 69.25
Cyclist AP_R40@0.50, 0.25, 0.25:
bbox AP:92.9363, 77.8973, 74.7406
bev AP:91.3753, 74.4753, 71.2521
3d AP:91.3753, 74.4753, 71.2521
aos AP:90.91, 73.02, 69.94
my result is as followed. Can't reach paper result as well.
Car AP@0.70, 0.70, 0.70:
bbox AP:96.3444, 89.7002, 89.3570
bev AP:90.2825, 88.2708, 87.8219
3d AP:89.3219, 79.1815, 78.6697
aos AP:96.29, 89.60, 89.20
Car AP_R40@0.70, 0.70, 0.70:
bbox AP:98.5869, 94.7842, 94.3787
bev AP:95.5221, 91.1952, 90.4819
3d AP:92.1301, 83.0165, 82.4530
aos AP:98.55, 94.65, 94.17
Car AP@0.70, 0.50, 0.50:
bbox AP:96.3444, 89.7002, 89.3570
bev AP:96.4119, 95.0143, 89.3356
3d AP:96.3548, 89.6420, 89.3110
aos AP:96.29, 89.60, 89.20
Car AP_R40@0.70, 0.50, 0.50:
bbox AP:98.5869, 94.7842, 94.3787
bev AP:98.6170, 96.5681, 94.6616
3d AP:98.5998, 94.7933, 94.5658
aos AP:98.55, 94.65, 94.17
Pedestrian AP@0.50, 0.50, 0.50:
bbox AP:73.9703, 69.3134, 65.1060
bev AP:67.1262, 62.2761, 58.3305
3d AP:65.2473, 58.9071, 54.1330
aos AP:69.99, 64.78, 60.57
Pedestrian AP_R40@0.50, 0.50, 0.50:
bbox AP:74.5213, 69.2424, 65.4731
bev AP:68.1703, 61.3948, 57.1665
3d AP:65.3005, 58.0700, 53.0599
aos AP:70.24, 64.47, 60.57
Pedestrian AP@0.50, 0.25, 0.25:
bbox AP:73.9703, 69.3134, 65.1060
bev AP:77.7762, 73.9847, 70.6240
3d AP:77.7708, 73.9738, 70.5758
aos AP:69.99, 64.78, 60.57
Pedestrian AP_R40@0.50, 0.25, 0.25:
bbox AP:74.5213, 69.2424, 65.4731
bev AP:80.1604, 75.3170, 71.0819
3d AP:80.1562, 75.2898, 71.0402
aos AP:70.24, 64.47, 60.57
Cyclist AP@0.50, 0.50, 0.50:
bbox AP:93.6250, 82.5444, 79.5846
bev AP:91.7165, 74.9170, 72.3592
3d AP:86.6990, 72.5479, 68.8231
aos AP:93.43, 82.14, 79.03
Cyclist AP_R40@0.50, 0.50, 0.50:
bbox AP:95.8189, 83.8564, 80.5940
bev AP:93.8289, 77.2388, 72.4899
3d AP:90.9577, 73.4490, 69.0680
aos AP:95.62, 83.41, 79.99
Cyclist AP@0.50, 0.25, 0.25:
bbox AP:93.6250, 82.5444, 79.5846
bev AP:92.7600, 79.5627, 76.4189
3d AP:92.7535, 79.5596, 76.4140
aos AP:93.43, 82.14, 79.03
Cyclist AP_R40@0.50, 0.25, 0.25:
bbox AP:95.8189, 83.8564, 80.5940
bev AP:95.2301, 80.5773, 77.2188
3d AP:95.2282, 80.5732, 77.1931
aos AP:95.62, 83.41, 79.99
Hi,
You can run it twice, or try to use OpenPCDet for reproducing.
I have added Voxel R-CNN into OpenPCDet now.
Best Regards,
Jiajun Deng
Hi! I have found that the VoxelRCNN needs trainging for a large epoch.
I trained it for 200 epoch, the result is much better.
Car AP@0.70, 0.70, 0.70:
bbox AP:96.5597, 89.5578, 89.2024
bev AP:90.2838, 88.2610, 87.8270
3d AP:89.4178, 83.9840, 78.7852
aos AP:96.48, 89.42, 88.99
Car AP_R40@0.70, 0.70, 0.70:
bbox AP:98.6332, 94.6860, 94.3009
bev AP:95.5642, 91.1385, 90.6047
3d AP:92.2998, 85.0574, 82.7184
aos AP:98.57, 94.50, 94.03
Car AP@0.70, 0.50, 0.50:
bbox AP:96.5597, 89.5578, 89.2024
bev AP:96.7073, 94.9496, 89.2635
3d AP:96.6240, 94.8539, 89.2159
aos AP:96.48, 89.42, 88.99
Car AP_R40@0.70, 0.50, 0.50:
bbox AP:98.6332, 94.6860, 94.3009
bev AP:98.6816, 96.6907, 94.5986
3d AP:98.6533, 96.2646, 94.4995
aos AP:98.57, 94.50, 94.03
Pedestrian AP@0.50, 0.50, 0.50:
bbox AP:74.5250, 69.4242, 65.4630
bev AP:69.0770, 62.4486, 58.6064
3d AP:65.8226, 59.1210, 55.0103
aos AP:70.85, 65.54, 61.31
Pedestrian AP_R40@0.50, 0.50, 0.50:
bbox AP:74.8678, 69.7596, 65.7239
bev AP:69.2748, 62.1164, 57.4054
3d AP:66.7230, 58.7470, 53.8872
aos AP:70.85, 65.36, 61.06
Pedestrian AP@0.50, 0.25, 0.25:
bbox AP:74.5250, 69.4242, 65.4630
bev AP:77.1194, 73.0611, 70.5424
3d AP:77.1091, 72.8356, 70.3701
aos AP:70.85, 65.54, 61.31
Pedestrian AP_R40@0.50, 0.25, 0.25:
bbox AP:74.8678, 69.7596, 65.7239
bev AP:79.5801, 74.6711, 71.3373
3d AP:79.5391, 74.3442, 70.5110
aos AP:70.85, 65.36, 61.06
Cyclist AP@0.50, 0.50, 0.50:
bbox AP:94.8819, 81.9523, 80.4192
bev AP:93.6120, 74.8324, 72.8478
3d AP:86.3606, 72.7165, 69.2049
aos AP:94.66, 81.54, 79.94
Cyclist AP_R40@0.50, 0.50, 0.50:
bbox AP:96.2394, 83.9695, 81.0132
bev AP:94.5897, 77.3344, 73.9650
3d AP:91.1998, 74.0525, 69.5397
aos AP:96.05, 83.52, 80.53
Cyclist AP@0.50, 0.25, 0.25:
bbox AP:94.8819, 81.9523, 80.4192
bev AP:93.4143, 79.1195, 76.6823
3d AP:93.4143, 79.1195, 76.6823
aos AP:94.66, 81.54, 79.94
Cyclist AP_R40@0.50, 0.25, 0.25:
bbox AP:96.2394, 83.9695, 81.0132
bev AP:94.8013, 80.2909, 77.0861
3d AP:94.8013, 80.290
I have never trained it for more than 80 epochs.
Have you used the same batch size setting as mine?
That's an interesting finding.
Thanks for your comments.
I modified the batch_size to 8.
The results of 300 epoch is not well as 200 epoch.
I didn't test on 80 epoch, but I will try.
Thank you for your work.
Hi,
You can run it twice, or try to use OpenPCDet for reproducing.
I have added Voxel R-CNN into OpenPCDet now.Best Regards,
Jiajun Deng
I use 8 GPU and batch size is 4*8 totally, and all setting is same as offical. And I use offical OpenPCDet for training. My result is as above.
I can't reach the results either. I only have 3 GPUs, so I can only try 8 & 12 batches.
For 12 batches, I trained for 260 epochs, and for 8 batches I trained for 180 epochs. I save a checkpoint every 10 epochs. The best evaluation result of Car AP_R40 moderate is near 83, just like @MengAaron
Car AP_R40@0.70, 0.70, 0.70:
bbox AP:97.9632, 94.4596, 92.2352
bev AP:93.3391, 90.7738, 88.7476
3d AP:91.6714, 82.7496, 80.2826
aos AP:97.93, 94.32, 92.04
Here is the plot for 8 batches and 180 epochs
And I tried smaller learning rate for 12 batches, which was not helpful either.
Is the save checkpoint interval too long? I have no idea