Test accuracy inquiry

Question

Test accuracy inquiry

Closed this issue 2 months ago · 6 comments

Hello, I have read your paper and would like to use it as a baseline. I would like to ask if I used the code and methods you provided on GitHub for training and testing, but it did not meet the expected level in your paper. For example, in 7scenes_fire, the test result I obtained was 5cm/5deg: 93.3%, Median Error: 0.88deg, 2.0cm， But in your paper, the result is 5cm/5deg: 99.5%, Median Error: 0.30 / 0.77， I would like to ask you what the reason is.

Answer 1 · 2024-08-30T12:45:05.000Z

Hi, what performance can you get with the ACE baseline?

Answer 2 · 2024-08-31T06:07:20.000Z

I tested the fire scene on the ACE baseline and found that it was also slightly lower, only 5cm/5deg: 91.2%. May I ask if this is a device issue? I am using the Nvidia 4070s.

Answer 3 · 2024-08-31T06:21:41.000Z

there might be some issues with the datasets, did you try the Cambridge dataset and see if you can get the same performance?

Answer 4 · 2024-08-31T09:07:11.000Z

Thank you for your reminder. I tried the Cambridge Kings College scene. The results obtained are Median Error: 0.3deg, 18.2cm, and 5cm/5deg: 4.4%

Answer 5 · 2024-08-31T09:12:32.000Z

That is comparable to what I obtained for the paper. I'd suggest checking the 7-scenes dataset source. Feel free to reply if you have any question.

Answer 6 · 2024-08-31T09:24:19.000Z

Okay, thank you very much for your suggestion and help！