Wrong image id to score associations
peteanderson80 opened this issue · 3 comments
Hi,
I noticed that COCOEvalCap
zip's the image ids and scores together so that individual scores can be accessed:
https://github.com/tylin/coco-caption/blob/master/pycocoevalcap/eval.py#L66
I think this is done incorrectly. The scores are ordered according to the output of multiple independent calls to dict.keys()
in each scorer, e.g.
https://github.com/tylin/coco-caption/blob/master/pycocoevalcap/cider/cider.py#L33
However I think dict.keys()
returns keys in an arbitrary order.
In my experience using the code, the associations in imgToEval
were wrong, which caused me some confusion. An easy fix would be to replace each call to dict.keys()
with sorted(dict.keys())
.
You seem right, and this can invalidate the results on the evaluation server.
I would suggest to replace assert(gts.keys() == res.keys())
with assert(sorted(gts.keys()) == sorted(res.keys()))
in all the metrics for the same reason.