Question about reproducing results

[smirchan]

Hi,

Thanks for providing pre-trained model weights. It appears from the README that we can run

export MODELS_DIR="./models/Shopping100k";
python src/eval.py --dataset_name ${DATASET_NAME} --file_root splits/${DATASET_NAME} --img_root ${DATASET_PATH} --load_pretrained_extractor ${MODELS_DIR}/extractor_best.pkl --load_pretrained_memory ${MODELS_DIR}/memory_best.pkl 

to reproduce the top-30 accuracy and NCDG of the paper after downloading the pre-trained weights.

I am instead seeing the following results:

Do you know what the problem could be?

[lorisbaz]

Thanks for your feedback and trying out our code. Can you please share some information about how you are running it? Do you run it on CPU? Did you install git lfs before cloning the package? How is your Shopping100k dataset folder organized?

I just re-run the evaluation on GPU and these are the results I got:

[smirchan]

Thanks for re-running the evaluation. This must be an issue on my end then.

I ran it on an A100 GPU. I was not able to install git-lfs (no sudo access), so I downloaded the models directly from GitHub and placed them in the right directories.

I set DATASET_PATH to be the ".../Shopping100k/Images" folder, which contains Female and Male subdirectories, and then category subdirectories within them.

[lorisbaz]

Have you been able to successfully run the experiment? Have you tried using git-lfs?

[smirchan]

I was unable to run install git-lfs (no sudo access) so I could only use the downloaded checkpoints. I never figured out why the evaluation results I was getting were lower, but I no longer need this experiment, so I'm happy to close the issue. Thanks!