Why don't the author compare the results of the paper with some top methods on KITTI leaderboard?
sisrfeng opened this issue · 5 comments
I' m not familiar with the dataset and hope that someone can explain more about the leaderboard.
Thank you!
on the COCO dataset [14], the AP of Gaussian YOLOv3 is 36.1, which is 3.1 higher than YOLOv3.
In particular, the AP 75 (i.e., strict metric) of Gaussian YOLOv3 is 39.0, which is 4.6 higher than that of YOLOv3.
Why isn't there a table comparing the results with other methods on COCO?
Are the models which are tested on KITT only trained on KITTI, not on BDD?
@sisrfeng
Hi,
In autonomous driving, both high accuracy and real-time detection speed are extremely important, but no published studies within top-10 of the KITTI page achieve a detection speed of above 20 fps when we submitted our paper on ICCV. For this reason, we compared the proposed algorithm with previous studies that not only achieve a high accuracy but also offer a fast detection speed in general datasets.
Our initial submitted paper for ICCV, we only presents the experimental results on the typical autonomous driving dataset, KITTI and BDD, in order to show the improvement of performance by focusing on autonomous driving (our contribution).
However, in rebuttal period, some reviewers mentioned the experimental result of COCO dataset. So, we added it in our camera-ready version. But, I couldn't put the table because of page limitation. So I mentioned it briefly in word.
In experiment, we only use each dataset on testing and training, respectively. We didn't add any other data.
@sisrfeng
Hi,
In autonomous driving, both high accuracy and real-time detection speed are extremely important, but no published studies within top-10 of the KITTI page achieve a detection speed of above 20 fps when we submitted our paper on ICCV. For this reason, we compared the proposed algorithm with previous studies that not only achieve a high accuracy but also offer a fast detection speed in general datasets.Our initial submitted paper for ICCV, we only presents the experimental results on the typical autonomous driving dataset, KITTI and BDD, in order to show the improvement of performance by focusing on autonomous driving (our contribution).
However, in rebuttal period, some reviewers mentioned the experimental result of COCO dataset. So, we added it in our camera-ready version. But, I couldn't put the table because of page limitation. So I mentioned it briefly in word.In experiment, we only use each dataset on testing and training, respectively. We didn't add any other data.
Thank you very much!
I can not find YOLOv3 and SSD on the KITTI leaderboard.
Q1: Is the reason that their mAP is not high enough though FPS is high?
On KITTI leaderboard, RFBNet' s run time is 0.2s.
Q2: Should the FPS be 5?
In your table, it is 39.2. Is the reason that they run on different devices?
Q3:Would you mind put the result comparsion on COCO here?
Q4：If I train the model on the combination of KITTI, COCO, UA-DETRAC and BDD, or even other dataset containing vehicle, do you think I can get better result on a single dataset like UA-DETRAC?
Q1, Q2: Please read the YOLO, SSD, and RFBnet paper in detail. RFBnet is very fast (mentioned the RFBnet paper). I don't know the result of RFBnet in KITTI leaderboard, because there is no explanation about environment in detail. We re-trained the RFBNet in our environment.
Q3: I mentioned the COCO results in our paper. You just compared it with other works.
Q4: I think you can get better result if you use many datasets, which has same view point.