KITTI ground truth labels provide poor quality
Closed this issue · 4 comments
Hi @darrenjkt,
Hope you are doing fine!
I have been using several detectors (pv_rcnn_plusplus and voxel_rcnn models trained on waymo,nuscenes and lyft) in order to label KITTI frames. Then, I noticed that, there are several frames that has missing ground truth boxes even though our detectors were able to label them correctly which made me curious if there is something I am doing wrong or KITTI ground truth is really bad?
Kind regards,
Gorkem
Yeah that's correct. From memory, KITTI ground truth labels do not label objects that are smaller than 25 pixel height. This tends to exclude far range objects that lidar can pick up which is why kitti lidar ground truth is missing labels.
Waymo and nuScenes on the other hand, will label any objects with at least one lidar point in them regardless of image pixel size. That's why detectors trained on them can detect objects with less points.
Hi @darrenjkt, thanks a lot for your reply!
Then, for the quantitative evaluation purposes; is it a good approach to discard these object somehow? Because this results in a lot of false positives which significantly affects the results.
Exactly, using the waymo dataset I was able to get reasonable results both quantitative and qualitative...
Best,
Not sure if the openpcdet kitti evaluation already does some filtering but you could dig into their code first.
If not, you could filter the predictions based on their 3d bounding box pixel height after projecting into the image frame.
Make sure you re-run this modified evaluation on other algorithms for fair comparison.
Alright, thanks for your help!