Performance improvement gap between the val and test set on ScanNet

Question

Performance improvement gap between the val and test set on ScanNet

Closed this issue a year ago · 1 comments

Hi, thanks for your great work on 3D instance segmentation. I have a question about the performance on the ScanNet dataset.

I noticed that your results on the test set improved by 3.2% on the AP50 metric compared with the val set results. However, when I tried my own model on the ScanNet benchmark, I found that my val set and test set results are very similar. Do you know why ISBNet performs much better on the test set? I would love to know how to improve my own models.

Looking forward to your reply, thanks in advance!

Answer 1 · 2023-06-24T08:34:01.000Z

I guess one reason is that the official test set comprises only 100 scenes, leading to more unstable results.