tylin/coco-caption

Wrong image id to score associations

peteanderson80 opened this issue · 3 comments

Hi,

I noticed that COCOEvalCap zip's the image ids and scores together so that individual scores can be accessed:
https://github.com/tylin/coco-caption/blob/master/pycocoevalcap/eval.py#L66

I think this is done incorrectly. The scores are ordered according to the output of multiple independent calls to dict.keys() in each scorer, e.g.
https://github.com/tylin/coco-caption/blob/master/pycocoevalcap/cider/cider.py#L33

However I think dict.keys() returns keys in an arbitrary order.

In my experience using the code, the associations in imgToEval were wrong, which caused me some confusion. An easy fix would be to replace each call to dict.keys() with sorted(dict.keys()).

You seem right, and this can invalidate the results on the evaluation server.

tylin commented

The dict.keys() is in arbitrary order but not in random.
The evaluation code uses gt.keys() to retrieve imgIds so it's fine now after the fix 1dcaa93
The leaderboard result is not affected. For details please see #11

I would suggest to replace assert(gts.keys() == res.keys()) with assert(sorted(gts.keys()) == sorted(res.keys())) in all the metrics for the same reason.