Lower values of recall@k than reported on the paper on the CUB dataset
Closed this issue · 6 comments
Hello, friend. Thanks for studying your research work. I tried the CUB-200-2011 Dataset's evaluation set and obtained the following results:
Recall at 1 = 66.9
Recall at 2 = 76.4
Recall at 4 = 85.7
Recall at 8 = 91.4
These values seem to be less than those reported on the paper, Table 1. What could be the reason for this?
The performance you get seems to be too low compared to the performance I reported in the paper.
Please check whether your current experimental setting is different from the paper's default setting.
For example, please check whether the batch size is set small and whether the weights of the pre-trained source model are properly loaded.
When I just ran the same code, I got a performance above 69 in 4 epochs as shown in the figure below.
If all settings are the same as ours, please show us your training graph or printout as I want to check the value of the loss.
I find a question.
After running this code, the number of images in the evaluation set is 3043 instead of 5794 after filtering the inappropriate images.
In ./code/dataset/cub.py
if y in self.classes and fn[:2] != '._': self.ys += [y] self.I += [index] self.im_paths.append(os.path.join(self.root, i[0])) index += 1
Perhaps you have a dataset file that doesn't match my code. The code mentioned is for excluding the files other than the folder, not the code that filters the images that are actually used. The total number of test images must be exactly 5924.
Please read the ReadMe file again and download the dataset from the link described.
Please also check whether you have the CUB-200 (also called CUB-200-2010) dataset, not the CUB-200-2011 dataset.
In CUB-200, the total number of datasets (train & eval) is about 6000, and the number of classes is 200.
Predictably, I think you used this dataset.
Dataset link: http://www.vision.caltech.edu/visipedia/CUB-200-2011.html
Yes. I used CUB-200-2011, maybe you could modify your CUB-200-2011 in readme file to CUB-200.
Thank you so much.
Oh sorry, there was a mistake in what I said.
To correct, what we used is CUB-200-2011 and I think the one you used is CUB-200.
When using CUB-200-2011, our code provides 5924 evaluation images.
Please check again whether the dataset you are using is CUB-200-2011.
OK, I'll check the dataset. Thank you so much.