GuangxingHan/QA-FewDet

Coco dataset recurrence problem

ntynitianyu opened this issue · 9 comments

ntynitianyu commented

Hello, the coco dataset cannot reproduce the performance of the paper, but the voc dataset can. Is this because my coco data set has an error?

GuangxingHan commented

Hi, thanks for your interest in our work. Do you mean the meta-testing results or fine-tuning results? Could you show me the training log to see what happened there? Have you tried our pre-trained model?

ntynitianyu commented

I mean the meta training result.this is the baseline model log of FewX
meta_training_coco_resnet101_stage_1.txt

GuangxingHan commented

Thanks for sharing the log. I find that one hyper-parameter (SUPPORT_SHOT during meta-training) in the config file is responsible for the inferior performance. Please use the default value 30. If you cannot make it due to the GPU memory limitation, please try the largest shot you can. A smaller SUPPORT_SHOT cannot extract accurate and stable support feature, leading to the performance degradation.

INPUT:
  FS:
    SUPPORT_WAY: 2
    SUPPORT_SHOT:  30 #5
ntynitianyu commented

Thanks for the answer, I will start an experiment to verify this problem.

ntynitianyu commented

I modified the hyper-parameter, but still can't get good performance。
meta_training_coco_resnet101_stage_1.txt

GuangxingHan commented

Thanks for your feedback. According to the log, the 5-shot meta-testing results is nAP=7.0267, which is exactly what we report as the baseline model in the paper. Please note that this is only the first stage training, and our QA-FewDet model needs the second stage training.
It is worth mentioning that, our current results using the exact default config file in this repo is slightly better than the paper reported. I notice that you use a smaller batch size (IMS_PER_BATCH), smaller lr (BASE_LR), and potential smaller training image sizes (MIN_SIZE_TRAIN), which can lead to slightly lower performance. You can double check these hyper-parameters in our default config file here.

ntynitianyu commented

Thanks for your answer. Did the results of the paper refer to nAP? I always thought it was an AP indicator, so I compared a wrong indicator

GuangxingHan commented

Actually we provided all of the results for nAP, nAP50, and nAP75. nAP is the most commonly used metric for COCO.
For 5-shot meta-testing of the baseline model (also the stage 1 model):

  • The results in our paper is 6.9, 14.3, 6.0 respectively.
  • The results in your training log is 7.0267,14.5090,6.0813.
ntynitianyu commented

Thanks for answering my question