Score gap for +1m datasets

Question

Score gap for +1m datasets

Closed this issue a year ago · 8 comments

I get +1m score based on resnet50 with a little gap from the paper.
Roxford
Without rerank
Retrieval results: mAP E: 87.98, M: 73.13, H: 50.75
With rerank
Retrieval results: mAP E: 92.11, M: 78.7, H: 62.36

Rparis
Without rerank
Retrieval results: mAP E: 90.6, M: 79.3, H: 62.62
With rerank
Retrieval results: mAP E: 92.71, M: 82.24, H: 68.0

Answer 1 · 2023-09-08T14:44:15.000Z

We will rerun the experiments on +1M datasets, stay tuned. Also, have you found whether the numbers on ResNet101 have gaps or not?

Answer 2 · 2023-09-08T14:57:50.000Z

And whether the setting you use here can reproduce the results on ROxf 5k and RPar 6k?

Answer 3 · 2023-09-09T01:33:18.000Z

I can reproduce the results on ROxf and Rpar without R1M, but there is a gap with distractor.

Answer 4 · 2023-09-09T05:07:50.000Z

I can reproduce the results on ROxf and Rpar without R1M, but there is a gap with distractor.

@maolaoshi301 Cool! My Google collaborators are re-verificating the results on 1M because 1M datasets are stored at their side. Should be quick. Stay tuned!

Answer 5 · 2023-09-12T02:31:24.000Z

I reproduce R101 on Rpar/Roxf+R1M, there is also a gap.
Roxford+R1M:
Retrieval results: mAP E: 89.48, M: 77.69, H: 60.44
Rparis+R1M
Retrieval results: mAP E: 92.7, M: 82.34, H: 66.6

Answer 6 · 2023-09-12T12:35:36.000Z

I reproduce R101 on Rpar/Roxf+R1M, there is also a gap. Roxford+R1M: Retrieval results: mAP E: 89.48, M: 77.69, H: 60.44 Rparis+R1M Retrieval results: mAP E: 92.7, M: 82.34, H: 66.6

Without reranking?

Google is re-verificating results and I am also working on my side to reproduce the results on 1M.

Answer 7 · 2023-09-12T13:22:01.000Z

Yeah, this score is without reranking.

Answer 8 · 2023-09-13T09:13:28.000Z

Hi! Thanks for your patience. I just finished reproduce the 1M results on RN101 on my side. Here are the details:

ROxford+1M:

RN101 w/o reranking:
Retrieval results: mAP E: 90.09, M: 78.69, H: 62.27

RN101 w/ reranking:
Retrieval results: mAP E: 94.72, M: 84.2, H: 71.27

RN50 w/o reranking:
Retrieval results: mAP E: 88.72, M: 74.24, H: 52.69

RN50 w/ reranking:
Retrieval results: mAP E: 92.66, M: 79.95, H: 64.12

RParis+1M:

RN101 w/o reranking:
Retrieval results: mAP E: 93.25, M: 83.46, H: 68.53

RN101 w/ reranking:
Retrieval results: mAP E: 94.36, M: 85.5, H: 72.47

RN50 w/o reranking:
Retrieval results: mAP E: 91.11, M: 80.53, H: 64.6

RN50 w/ reranking:
Retrieval results: mAP E: 92.94, M: 83.23, H: 69.23

At this stage, you can see my reproduction is significantly different from yours, and is very close to the paper results (well, even slightly better, probably due to some implementation details between me and google). I plan to release the evaluation code for 1M as well in the next few days for you to check whether you or I made some mistakes. For now, I stongly encourage you to double check your reproduction code. And a probably tip would be, to firstly reproduce the 1M results on CVNet vanilla weights to see if your pipeline is correct, since we have checked this pretrained weight and it can reproduce the exact results on their paper. I will remain this issue open for now, until our full reproduction results and 1M reproduction code is released.

UPDATE 9/14/2023: Since all the reproduction results and evaluation codes were released, I closed this issue.