drprojects/superpoint_transformer

the number of fold

Closed this issue · 3 comments

Charlie839242 commented

Hi Damien,

Thanks for your great paper and awesome project!

I ran the command "python src/train.py experiment=s3dis datamodule.fold=5" many times but still cannot reproduce the result. Do I need to change it to "python src/train.py experiment=s3dis datamodule.fold=6"?

Looking forward to your reply!

Charlie839242 commented

Besides, when I use datamodule.fold=6, I simply obtain mIOU≈81%. Is there something wrong?

drprojects commented

Hi @Charlie839242, thanks for your interest in the project.

Correct me if I am wrong but it sounds like you may be confusing the S3DIS Area-5 metrics and the S3DIS 6-fold metrics. When you run:

python src/train.py experiment=s3dis datamodule.fold=X

You train a model on fold X. Meaning you keep Area-X as test set and train on all the other areas.

So, for fold-5 (ie Area-5 as test set), you should normally get something close to 68.9 mIoU as stated in our paper. It may be a little lower like 68.0, since we kept the best out of 3 trainings, but not much lower.

The 6-fold metrics, on the other hand, are the cross-validation performance for the 6 models trained on each fold separately. This is a standard procedure in the literature for semantic segmentation on S3DIS.

Charlie839242 commented

Thanks for your clear explaination! I will close this issue.

  • 👍1