Issue with ASpanFormer's performance on ScanNet

[LinKInLeLe43]

Hi, thanks for your excellent work!

I evaluated ASpanFormer on ScanNet and achieved significantly higher accuracy: 22.4/41.0/57.9, compared to the 19.6/37.7/54.4 reported in Table 1. Additionally, after shuffling and averaging five results, I still obtained 21.8/40.5/57.3. Since ASpanFormer does not report this cross-domain result and has no open issues on GitHub, I would appreciate any insights you might have.

[wyf2020]

Thank you for your question! Our cross-domain results for ASpanFormer were obtained by running the indoor script of ASpanFormer, where the ckpt needed to be changed to the outdoor version and the NPE res for training and testing in the indoor config file should be adjusted accordingly. We tested this using the environment provided by ASpanFormer, which includes python=3.8, pytorch=1.8.1 and cudatoolkit=10.2, on a V100 GPU and we performed RANSAC evaluation once.

[LinKInLeLe43]

Thanks for your reply!

I conducted a quick validation and found that this may be caused by the NPE resolutions as you mentioned. I reported the cross-domain results below for EfficientLoFTR and ASpanFormer using fixed training resolutions of [832, 832] and varied test resolutions.

I am trying to confirm that when the test resolutions of ASpanFormer are set to [480, 640], the results are closer to the metrics reported in your paper. However, EfficientLoFTR sets the test resolutions to [832, 832]. Since using the same test resolutions for ASpanFormer also results in higher accuracy, is there an issue with inconsistent test settings here?

	[832,832]	[640,640]	[480,640]
EfficientLoFTR	18.9/36.6/53.4	18.5/36.2/52.8	18.2/36.0/52.8
ASpanFormer	22.4/41.0/57.9	21.4/39.3/56.2	20.1/38.6/55.4