Evaluation file eval.lua

[yauhen-info]

Regarding https://github.com/jiasenlu/HieCoAttenVQA/blob/master/eval.lua, is it possible to use it for VQA evaluation?

thank you.

[xhzhao]

@YauhenMinsk I don't know, because i did not reach the target accuracy on the repo. I suggest you to use this file: https://github.com/VT-vision-lab/VQA_LSTM_CNN/blob/master/eval.lua , and i got the target accuracy on test-dev2015(62.38% for MultipleChoice, 57.63% for OpenEnded)

[yauhen-info]

@xhzhao , thanks for advice.

[panfengli]

The eval.lua works well, it will generate a json file storing the results. Added the evaluation codes provided by VT-vision-lab/VQA, an example result showed as:

Overall Accuracy is: 47.49

Per Question Type Accuracy is the following:
are there : 62.48
what brand : 33.27
what room is : 81.82
what color is : 43.87
is : 63.62
are they : 64.61
what number is : 4.43
what does the : 18.67
is this person : 63.41
is the : 65.01
what is the man : 42.74
how many : 37.05
does this : 67.42
is there a : 63.31
are : 62.71
is he : 65.60
what : 32.41
does the : 68.33
is the person : 63.21
where is the : 23.13
what animal is : 61.50
how : 21.13
is the man : 64.69
what is the woman : 36.27
none of the above : 47.78
which : 38.85
where are the : 26.54
are the : 64.77
how many people are : 37.25
what is on the : 28.73
has : 65.74
was : 65.48
what type of : 40.38
is this an : 67.12
do : 68.81
can you : 61.71
who is : 21.17
are these : 64.77
what are : 37.92
what is the : 32.95
is the woman : 63.71
do you : 66.78
what time : 21.09
what are the : 32.13
what color are the : 45.14
why : 14.49
what is this : 44.78
what is the person : 45.22
how many people are in : 37.40
is this : 64.52
why is the : 13.42
what is the color of the : 53.54
what is : 26.54
could : 77.02
are there any : 61.68
what is in the : 32.09
what sport is : 73.56
what kind of : 40.49
is it : 69.30
is this a : 65.07
what is the name : 8.15
is there : 64.09
what color is the : 47.83
what color : 37.46
is that a : 62.42


Per Answer Type Accuracy is the following:
other : 36.98
number : 32.15
yes/no : 66.64


ground truth answers
Question: What two animals are photoshopped together?
Answer 1: elephant and sea lion
Answer 2: elephant and penguins
Answer 3: elephants
Answer 4: yes
Answer 5: yes
Answer 6: walrus and elephant
Answer 7: elephant and penguins
Answer 8: elephant and seal
Answer 9: penguins and elephant
Answer 10: elephant penguin


generated answer (accuracy 30.00)
Answer:   elephants