BradyFU/DVG

Some issues on Evaluate Rank-1 and ROC

ZYBOBO opened this issue · 2 comments

ZYBOBO commented
  1. Get probe and gallery list.
  2. Evaluate Rank-1 and ROC.
    2.1 For rank-1 accuracy, you can select each probe image and compute the cosine similarity between it and all the gallery images.
    2.2 For ROC, you can compute all the cosine similarity between probe and gallery (total about 200w+ pairs), and then feed them into a roc curve api such as sklearn.metrics.roc_curve.

Hello, from your reply to #5, I have some questions for your help.

  1. From your train_lightcnn.py, Rank-1 accuracy is not the cosine similarity beteween probe image and gallery image. Is there anything different between train accuracy and validation accuracy?
  2. Do you mean that VR@FAR is the point from ROC? Can I regard VR@FAR as TPR@FPR?
  3. How do you determine the corresponding threshold when FAR is 1% or 0.1%?

I'm looking forward for your kind reply. Thanks~

Originally posted by @ZYBOBO in #5 (comment)

AlfredXiangWu commented

  1. In training phase, the top-1 accuracy is the classification accuracy. It is different from the rank-1 accuracy in face identification testing. Note that face recognition evaluations, including face verification and face identification, are open-set problem. The classes of testing images are exclusive to the training ones.

  2. VR@FAR is the same as TPR@FPR / TPR@FAR.

  3. For example, if given 2,000 negative pairs, Considering FAR=1%, we sort the scores of negative pairs in descending order and select top 20 (2,000*1%) scores. We can regard the 20th score as the threshold.

ZYBOBO commented

Thank you very much for your detailed answers~
Could you provide me an example for your validation data_list? Just like the probe and the gallery list, I wonder how to obtain the corresponding positive pairs and negetive paris. Do you use each probe vis images to pair all the corresponding gallery nir images? Meanwhile, paired-images with same ID between vis and nir mean positive pairs, nor mean negative pairs? I don't know how to use validation_list to build paired-images.
I hope you can help me in the above issue. Thanks again for your kind help~
Wish you good luck!