loss doesn't decrease

[Haerxu]

Hi,

I want to apply openscene in my own dataset. I follow the data preprocess instruction. However, the distillation loss always fluctuate around 0.2, like this:

And the validation iou is around 0:

The distillation training script is adapted from scannet. For hyper-parameter setting, I only changed batch size from 8 (original in scannet) to 4 in order to fit the gpu memory. I suspect it is because of the different batch size since the learning rate is also related to batch size:

Do you know how to solve this problem?

---

Also, I test the performance by using 2d features to represent point cloud. The results are okay (miou=0.2391).

[pengsongyou]

Emmmm that is strange. Could you try training with just 1 scene instead of the entire dataset, and see if your network is able to overfit this scene (i.e., getting similar results to 2D fusion results like miou=0.2391)?

[Haerxu]

Thanks for the reply. I found a bug in my 3d data processing code. It can produce reasonable performance now.

By the way, does batch size affect the performance a lot? When I reproduce the results for Scannet, I set the batch_size=4 to fit in the 24GB GPU. But the miou of 3D distillation model is around 42 (46 in the table 4 of paper).

[pengsongyou]

Sorry for the late reply, yes, batch_size makes some differences, so maybe you can consider using multi-GPU training to increase the batch_size?

Best
Songyou