Question regarding evaluation function (minor bug?)

[FabianHoerst]

Hello,

i think there might be a bug in the evaluation code or I might missunderstand something: In the evaluation function, first, the tuple is [x,y,z,label,score]. However, for creating the pred_dict, it is referenced as [x,y,z,score,label]. Why is the label used as score here?

MIDOGpp/evaluation.py

Lines 86 to 94 in c2726b1

	filtered_predictions = [(x,y,0) for x,y,z,cls,sc in self.predictions[case] if cls==1 and sc > det]
	bbox_size = 0.01125 # equals to 7.5mm distance for horizontal distance at 0.5 IOU
	pred_dict = [{'boxes': Tensor([[x-bbox_size,y-bbox_size, x+bbox_size, y+bbox_size] for (x,y,z,_,_) in self.predictions[case]]),
	'labels': Tensor([1,]*len(self.predictions[case])),
	'scores': Tensor([sc for (x,y,z,sc,_) in self.predictions[case]])}]
	target_dict = [{'boxes': Tensor([[x-bbox_size,y-bbox_size, x+bbox_size, y+bbox_size] for (x,y,z) in self.gt[case]]),
	'labels' : Tensor([1,]*len(self.gt[case]))}]

[maubreville]

Hey Fabian.

Thanks for that. We'll look into it.

Best,

Marc

[FraukeWilm]

Dear Fabian,

Yes, this was indeed a bug in our evaluation code which influenced the mAP calculation. Thank you for brining this to our attention. We have made the necessary code changes and will prepare an addendum for our manuscript. Fortunately, all evaluations in the manuscript were based on the F1 score, which was not affected by the bug.

Kind regards,
Frauke

[FabianHoerst]

Thank you for the response!

Sorry for the problem it may have caused! Luckily, your mAP results should get even better with the change.
Best, Fabian