The calculation method of mAP in object detection experiment
xiahongchi opened this issue · 4 comments
Hi, I'm quite interested in your work of SLidR and I try to reproduce your experiment of detection using OpenPCDet. But I'm confused about the method you use for calculating mAP, like the threshold setting, or the outputs of the test program you use for calculating. I would be appreciated if you can kindly answer my questions. Thanks!
Hi, we rely entirely on the code of OpenPCDet for the metrics in object detection. We haven't modified the code in any way that should modify the results given by OpenPCDet, and use the results with 40 recall positions as is now the default for KITTI detection. Maybe your confusion comes from the code of semantic segmentation that we provide, that is not used during downstream in object detection.
Now I've known that you use AP_R40, but for each classification, there're two 4*3 AP_R40 tables that show the results in the outputs of OpenPCDet. Can you tell me how you organize these data and calculate the final mAP results? I'm so sorry that I can't find the standard mAP calculation procedure for this specific KITTI dataset and from your paper I can't find it neither. It would be better if you can kindly post the link or paper that explain the standard calculating procedure for mAP of the KITTI dataset.
The printout of OpenPCDet is extremely confusing, but as far as I understand, the "official" KITTI detection metric corresponds to the 3d mAP of each class @r40 with threshold of 0.7 for the cars and 0.5 for the other classes, which corresponds to the table "Car AP_R40@0.70, 0.70, 0.70:" at the line "3d" the middle value being the moderate difficulty we report for instance for the class car.
Let's take this example :
Car AP@0.70, 0.70, 0.70:
bbox AP:95.2752, 89.5233, 89.2069
bev AP:90.4053, 88.3227, 87.8464
3d AP:89.5335, 83.0361, 78.9593
aos AP:95.24, 89.45, 89.08
Car AP_R40@0.70, 0.70, 0.70:
bbox AP:98.0047, 94.2789, 93.8967
bev AP:94.8042, 90.8358, 89.9937
3d AP:92.0217, 84.6512, 82.5493
aos AP:97.98, 94.18, 93.72
Car AP@0.70, 0.50, 0.50:
bbox AP:95.2752, 89.5233, 89.2069
bev AP:95.3162, 93.6363, 89.2336
3d AP:95.2777, 93.5749, 89.2008
aos AP:95.24, 89.45, 89.08
Car AP_R40@0.70, 0.50, 0.50:
bbox AP:98.0047, 94.2789, 93.8967
bev AP:98.0634, 96.0904, 94.2029
3d AP:98.0463, 95.5983, 94.1168
aos AP:97.98, 94.18, 93.72
Pedestrian AP@0.50, 0.50, 0.50:
bbox AP:71.0831, 66.7133, 64.5998
bev AP:65.2037, 60.4853, 55.5217
3d AP:63.5676, 57.6075, 53.2246
aos AP:66.29, 61.22, 58.84
Pedestrian AP_R40@0.50, 0.50, 0.50:
bbox AP:72.1530, 67.7733, 64.7285
bev AP:65.8177, 59.5897, 55.2454
3d AP:63.0918, 56.6227, 51.9996
aos AP:66.69, 61.36, 58.19
Pedestrian AP@0.50, 0.25, 0.25:
bbox AP:71.0831, 66.7133, 64.5998
bev AP:77.8360, 74.0713, 71.6428
3d AP:77.8189, 73.9873, 71.5448
aos AP:66.29, 61.22, 58.84
Pedestrian AP_R40@0.50, 0.25, 0.25:
bbox AP:72.1530, 67.7733, 64.7285
bev AP:79.5494, 75.5732, 72.2138
3d AP:79.5267, 74.9724, 71.6415
aos AP:66.69, 61.36, 58.19
Cyclist AP@0.50, 0.50, 0.50:
bbox AP:89.0580, 82.9940, 80.1540
bev AP:86.4252, 73.9092, 69.9174
3d AP:84.9780, 72.2635, 65.7067
aos AP:88.88, 82.18, 79.36
Cyclist AP_R40@0.50, 0.50, 0.50:
bbox AP:93.4500, 84.2134, 80.7097
bev AP:90.4235, 75.1635, 70.4838
3d AP:89.7646, 72.0706, 67.3153
aos AP:93.23, 83.36, 79.83
Cyclist AP@0.50, 0.25, 0.25:
bbox AP:89.0580, 82.9940, 80.1540
bev AP:90.0567, 79.5041, 76.5315
3d AP:90.0540, 79.5041, 76.5266
aos AP:88.88, 82.18, 79.36
Cyclist AP_R40@0.50, 0.25, 0.25:
bbox AP:93.4500, 84.2134, 80.7097
bev AP:93.3787, 80.3143, 76.8876
3d AP:93.3779, 80.3130, 76.8863
aos AP:93.23, 83.36, 79.83
Here the score would be 84.6512, 56.6227, 72.0706 for respectively the cars, pedestrians and cyclists
Now I fully understand the method you use for calculation, thanks for your answers! Best wishes!