Reproducing Results on DETRAC-Train Dataset with Mask-RCNN
siyuanc2 opened this issue · 3 comments
Hello,
I was trying to reproduce your results for the IOU-tracker on DETRAC-Train using detector Mask-RCNN. Although I have used the provided detection results and a new clone of your tracker, I keep getting results with MOTA of about 29.4 as opposed to the reported value of 34.36.
Tracker iou-tracker-master + Detector frcnn by DETRAC-MOT Evaluation:
PR-Rcll PR-Prcn PR-FAR|PR-MT PR-PT PR-ML|PR-FP PR-FN PR-IDs PR-FM| PR-MOTA PR-MOTP PR-MOTAL
34.8 41.0 0.36|28.8% 11.7% 26.7%|30139.2 187059.4 1006.6 995.2| 29.4 38.4 29.6
Would you mind looking at my attached logs below and see if I did anything wrong? Thank you very much for your help!
log.txt
Hi, I was finally able to reproduce your results. The reason is that apparently something goes wrong in the filtering process of the bounding boxes (dropping boxes from the ignored regions) in the detrac toolkit. If you run the evaluation on the raw detections provided in the download link you get indeed the results you posted. But the toolkit also reports a PR-MODA value of 62.96% for the detections, which is also way off.
BUT if you run the detector evaluation, you get a PR-MODA of 80.53%. The filtered detections are written back to the input files (MVI_*_Det_frcnn.txt...)
If you run the tracking evaluation again you you get the right tracking results and a PR-MODA of 80.22%. (which seems close enough to the 80.53% of the detector evaluation, note that the threshold steps are a bit different).
So the toolkit seems to work a bit different as expected. You need to run the detector evaluation script at least once to get the proper preprocessing of the detections, then you can evaluate the tracker.
I wonder if thats the reason for the worse results on the test set.
This was indeed the issue... After running the detector evaluation I was able to reproduce the results! Thank you very much for pointing this out!
my train reproduce
Tracker iou-tracker + Detector frcnn by DETRAC-MOT Evaluation:
PR-Rcll PR-Prcn PR-FAR|PR-MT PR-PT PR-ML|PR-FP PR-FN PR-IDs PR-FM| PR-MOTA PR-MOTP PR-MOTAL
43.4 46.8 0.64|37.4% 11.4% 24.0%|53716.1 170699.7 1034.2 985.6| 33.9 45.4 34.1
```