Unable to reproduce results on widerface using tinaface
manisoftwartist opened this issue · 3 comments
I am trying to reproduce the widerface results furnished here: https://github.com/Media-Smart/vedadet/tree/main/configs/trainval/tinaface.
I prepared the data as suggested in the Data Preparation
section. I also did the filtering step using python configs/trainval/tinaface/filter_widerface_val.py
. With the pretrained model of R50-FPN-BN
, I ran the evaluation script configs/trainval/tinaface/test_widerface.py
. Here are my results:
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 3226/3226, 0.5 task/s, elapsed: 5890s, ETA: 0s
+-------+-------+----------+--------+-------+
| class | gts | dets | recall | ap |
+-------+-------+----------+--------+-------+
| face | 31957 | 65363592 | 0.995 | 0.916 |
+-------+-------+----------+--------+-------+
| mAP | | | | 0.916 |
+-------+-------+----------+--------+-------+
From my results above, the mAP matches with your published results. But,
- The number of detections I get is
dets=65363592
. Why is this difference indets
not affecting the mAP? - Why do I get more detections in spite of the filtering step?
- I do not see the AP for easy, medium and hard subsets. How do I get these results?
- This evaluation has taken approximately 98 mins to complete on 3226 widerface validation images on GeForce GTX1080. Is there a way I can improve the speed?
- It is the characteristic of mAP metric.
- You mean you have more gt nums after the filtering step?
- Please refer to this for evaluation by using official tools.
- You can limit nms_pre, max_per_img, etc to limit output num and speed up.
@hxcai Thanks for your quick reply. My gts
seems to be correct (same as mike's comment in #24).
Here is my mAP using tinaface_r50_fpn_bn
+-------+-------+----------+--------+-------+
| class | gts | dets | recall | ap |
+-------+-------+----------+--------+-------+
| face | 31957 | 65363592 | 0.995 | 0.916 |
+-------+-------+----------+--------+-------+
| mAP | | | | 0.916 |
+-------+-------+----------+--------+-------+
And when using tinaface_r50_fpn_gn_dcn
+-------+-------+----------+--------+-------+
| class | gts | dets | recall | ap |
+-------+-------+----------+--------+-------+
| face | 31957 | 10358268 | 0.995 | 0.923 |
+-------+-------+----------+--------+-------+
| mAP | | | | 0.923 |
+-------+-------+----------+--------+-------+
So, the mAP on the easy, medium and hard subsets are obtained using the official widerface eval tools. So we need matlab to get these numbers!? Right?
@manisoftwartist Yeah, we run the official matlab code to get result.