matteorr/coco-analyze

KeyError for analyzing 2017 dataset

Closed this issue · 2 comments

kampta commented

I ran run_analysis.py on output of 2017 keypoint challenge and got this error

Traceback (most recent call last):
  File "run_analysis.py", line 127, in <module>
    main()
  File "run_analysis.py", line 115, in main
    paths = occlusionAndCrowdingSensitivity( coco_analyze, .75, saveDir )
  File "/home/kamal/dev/coco-analyze/analysisAPI/occlusionAndCrowdingSensitivity.py", line 49, in occlusionAndCrowdingSensitivity
    total_gts += overlap_index[no]
KeyError: 7

Has something changed from 2014 keypoint challenge?

mkocabas commented

Hi @kampta, I got same error as you. Did you solve the issue?

matteorr commented

@kampta, @mkocabas: The error was due to a change in the COCO keypoints annotations from version 2014 to version 2017 and is solved by this commit.

The variables

IOU_FOR_OVERLAP = .1
overlap_groups  = [[0],[1,2],[3,4,5,6,7,8]]
num_kpt_groups  = [[1,2,3,4,5],[6,7,8,9,10],[11,12,13,14,15],[16,17]]

are defined for analyzing the sensitivity of performance to occlusion and crowding on benchmarks of the 2014 version of COCO. By combining every list for the number of overlapping annotations (overlap_groups) with every list for the number of visible keypoints (num_kpt_groups) you obtain 12 benchmarks. The specific grouping was defined so that the number of annotations in each benchmark was as close as possible.

The variables overlap_index and keypoints_index are dictionaries that respectively contain the list of all the ids of the annotations in the COCO dataset that have a certain number of overlapping annotations (with IoU > .1) and a certain number of visible keypoints.

The error at this line in occlusionAndCrowdingSensitivity.py occurs because the code is appending to the list total_gts all the annotations that have 7 other overlapping annotations (with IoU > .1) but there are none of them
in overlap_index because there are none in COCO 2017, hence the missing key for value 7.

So, while COCO 2014 had some annotations with 7 overlaps with IoU > 0.1, COCO 2017 does not. As a final note, you should keep in mind that the specific values for the overlap_groups and num_kpt_groups are arbitrary and you can change them to fit your needs and the analysis code will still work.