Bad performance on ObjectNet

Question

Bad performance on ObjectNet

Opened this issue 2 years ago · 13 comments

I run your code and got a similar result with the paper except ObjectNet. The top-1 acc on ObjectNet is only 54.7%. I found the images in my ObjectNet is 50273 but not 50000. I downloaded the ObjectNet from the official url https://www.dropbox.com/s/raw/cxeztdtm16nzvuw/objectnet-1.0.zip and unzip it. Do you have any idea why this happened?

Answer 1 · 2023-06-07T15:10:50.000Z

Hi,
Thanks for pointing this out. I just checked my dataset, and it has 23.7k images only, probably some issue would have happened when I downloaded. I will update the paper with the correct numbers for ObjectNet once I redownload and re-evaluate.
This issue would be their for all the baselines as well. So shouldn't affect any conclusions or ordering of various finetuning approaches.
Thanks again for pointing this out.

Answer 2 · 2023-06-11T07:22:59.000Z

Really glad receive your reply! Another question I found is that when I evaluate on ObjectNet, not all images but only 18574 images of 113 classes are utilized. Is there something wrong with the code in ObjectNet?

Answer 3 · 2023-06-30T23:55:30.000Z

The code uses ImageNet data loader from folders (https://pytorch.org/vision/stable/generated/torchvision.datasets.ImageFolder.html). Can you check how many class folders are there in the local data path?

Answer 4 · 2023-07-03T06:42:24.000Z

There are 50273 images of 313 class folders in the local data path. Hope to see your revised results in your paper! Thanks!

Answer 5 · 2023-07-15T03:27:21.000Z

Hi, @SachinG007

I also have the same problem. If you already have any scores related to this, would you mind sharing them here? Thank you so much.

Answer 6 · 2023-07-15T08:44:02.000Z

Hi @BierOne ,
I haven't yet got a chance to rerun the numbers. I can try though getting numbers by late next week.

Answer 7 · 2023-07-15T08:48:22.000Z

Glad to hear that! Thanks :)

Answer 8 · 2023-07-29T12:07:24.000Z

Have you rerun the objectnet experiments yet? I got an acc of 55.4% for ZSL and 54.7% for FLYP. I can't match the performance of other methods (i.e. FT, LP, LP-FT), too.

Answer 9 · 2023-08-02T11:04:31.000Z

@LixDemon May I know if you could share your trained weight on ImageNet? It seems this repository does not provide it. Thank you!

Answer 10 · 2023-08-05T01:02:56.000Z

@LixDemon , I didn't yet get a chance to rerun the ImageNet experiments. I will get back on this. @Simon4Yan , I can upload the model once I rerun it.

Thanks

Answer 11 · 2023-08-13T20:25:14.000Z

Hi @LixDemon and @Simon4Yan ,
I am uploading a FLYP finetuned (on ImageNet) checkpoint of CLIP (ViT-B-16) here.

The updated FLYP and Zeroshot numbers for ObjectNet:
Zeroshot:
Evaluating on ObjectNet
ObjectNet Top-1 accuracy: 0.5333

Finetuned:
Evaluating on ObjectNet
ObjectNet Top-1 accuracy: 0.5499

I will also update the paper once I get the numbers for the baselines as well.
Thanks.

Answer 12 · 2023-08-14T01:26:21.000Z

Hi @SachinG007,
Really glad to hear that! Can't wait to see the revised paper.
Thanks for your detailed reply!

Answer 13 · 2023-08-14T01:28:51.000Z

@SachinG007 Thank you for sharing the model weights. Also, I am interested in the revised version. Kind Regards!