Unable to reproduce full Imagenet accuracies of pretrained weights for CMC Resnet50v2 and MoCo

[ajtejankar]

Hi @HobbitLong,

Thanks for such a clean and readable code.

I am interested in using the pre-trained weights that you were kind enough to provide. I downloaded the pre-trained weights CMC_resnet50v2.pth and MoCo_softmax_16384_epoch200.pth. Then, I ran the linear evaluation code with the following commands, but couldn't reproduce the accuracies. The accuracies at the final, 60th, epoch for CMC and MoCo are 62.0% and 57.3% respectively. The accuracies should be 64.1% (from the CMC paper) and 59.4% (from readme).

CUDA_VISIBLE_DEVICES=9 python LinearProbing.py --dataset imagenet \
 --data_folder /datasets/imagenet_nfs1 \
 --save_path ./output/cmc_linear \
 --tb_path ./output/cmc_linear \
 --model_path ./pretrained/CMC_resnet50v2.pth \
 --model resnet50v2 --learning_rate 30 --layer 6

CUDA_VISIBLE_DEVICES=8 python eval_moco_ins.py --dataset imagenet \
 --data_folder /datasets/imagenet_nfs1 \
 --save_path ./output/moco_linear \
 --tb_path ./output/moco_linear \
 --model_path ./pretrained/MoCo_softmax_16384_epoch200.pth \
 --model resnet50 --learning_rate 30 --layer 6

Have I missed something? Do I need to change the default hyperparameters to get the reported numbers?

Thanks

[HobbitLong]

Hi, @ChigUr ,

The hyper-parameters look correct to me.

The curve (for 60 epochs) on my end:
(1) CMC (I suppose this one is likely to be correct as the model was released several months ago)

(2) MoCo

Any other people run into the same issue?

[ajtejankar]

Hi @HobbitLong,

I've attached my test accuracy curves for your reference. It seems the problem my be specific to my setup. Let me investigate.

(1) CMC

(2) MoCo

Thanks

[HobbitLong]

Hi @ChigUr, how is things going? Should I close this issue?

[ajtejankar]

Sure. I haven't figured out the problem yet but it's likely to be specific to my setup.

[HobbitLong]

Ok, I will close it, but feel free to reopen.