Question about results on s3dis without colors
Closed this issue · 2 comments
Hello,
I am currently attempting to reproduce the results presented in your paper regarding the S3DIS dataset without using color information. According to your findings, the omission of color data resulted in a mean Intersection over Union (mIOU) decrease of only 2.5 points. Did you try to conduct this experiment with either the pc_tiling or xy_tiling methods activated.
Due to my computing machine, I need to activate this tiling my mIOU dropped approximately of 10 points, yielding an mIOU around 60,
Is this relatively drop expected?
While I understand that the absence of radiometric features would naturally lead to a decline in mIOU, I am unsure about the expected magnitude of this impact.
Sincerely,
Pierrick
Hi @gardiens, indeed removing radiometric information typically results in a performance drop.
As you noticed, the performance drop we communicate when removing RGB from pointwise features on S3DIS 6-fold is near 2.7 mIoU. However, this specific ablation only removes colors at training time, but still uses the same partition as the other ablations in the table. Said otherwise, the RGB color was used when computing the partition but not used as a point attribute when training. I agree this is a bit misleading.
Yet, in a previous paper where we worked on mutimodal learning on S3DIS, we did run the no-RGB ablation with a MinkowskiNet backbone on S3DIS folds 2 and 5 (https://arxiv.org/pdf/2204.07548). We did observe performance drops close to -10 mIoU, which seems to align with what you are observing too. Besides, it is possible that SPT suffers more from the lack of radiometric information than MinkowskiNet, since the partition construction then only relies on local, handcrafted features which may fail to capture all nuances of the local geometries.
Best,
Damien
Hi @gardiens, indeed removing radiometric information typically results in a performance drop.
As you noticed, the performance drop we communicate when removing RGB from pointwise features on S3DIS 6-fold is near 2.7 mIoU. However, this specific ablation only removes colors at training time, but still uses the same partition as the other ablations in the table. Said otherwise, the RGB color was used when computing the partition but not used as a point attribute when training. I agree this is a bit misleading.
Yet, in a previous paper where we worked on mutimodal learning on S3DIS, we did run the no-RGB ablation with a MinkowskiNet backbone on S3DIS folds 2 and 5 (https://arxiv.org/pdf/2204.07548). We did observe performance drops close to -10 mIoU, which seems to align with what you are observing too. Besides, it is possible that SPT suffers more from the lack of radiometric information than MinkowskiNet, since the partition construction then only relies on local, handcrafted features which may fail to capture all nuances of the local geometries.
Best, Damien
Thank you for your quick reply!