rafaelpadilla/review_object_detection_metrics

Incorrect mAP when categories are not used

Closed this issue · 4 comments

spamekkel commented

Hi,

Thanks for sharing your code. I have tried this and found a bug in the mAP calculation when the 'categories' in the annotations.json file contain more then used in the annotations. For example, in the tube test annotations.jon I've added 'frog'. Initially, the mAP is 1.0: predictions are 100% correct. But when 'frog' is added to the categories, the mAP is 0.66.

"categories": [
{
"id": 0,
"name": "cat"
},
{
"id": 1,
"name": "dog"
},
{
"id": 2,
"name": "frog"
}

Could you please check your code ?

rafaelpadilla commented

Hi @spamekkel, 😄

Thank you for your comment.

Based on the few details you provided, I believe there is no issue with the code. Mean Average Precision is computed by averaging the Average x Precision over all categories (classes) of your dataset.

Initially you have 2 classes and a mAP of 1. When you add a new class (frog), your results will be dropped by 1/3, since you made no detection in this new class.

Is that the case? I am missing anything here? 🤔 Please, let me know if I understood correctly.

spamekkel commented

When calculating mAP, if a certain class has no ground truth or predictions, it essentially means that the average precision (AP) for that class is undefined.

The class with no ground truth or predictions is typically excluded from the mAP calculation. mAP is the mean of the APs of all classes, so if a class has no data, it doesn’t contribute to the mean. This means the mAP is calculated over the remaining classes that have valid AP values.

rafaelpadilla commented

I see your point now. No ground-truth nor detection was added, only the label of the category was included.

Makes total sense not to consider such category in the computation of the mAP. Thank you for raising that.

I will try to find some time to fix that issue. If you please, have free time and want to contribute with this repo, would you please open a PR with this fix? 🙏

github-actions commented

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.