CUDA out of memory

[Trulli99]

Hello, thank you in advance for your work.

When running the part segmentation I face an error of CUDA out of memory. I saw in a comment of other issue about this error that at the peak the model uses ~19GB of memory. I am using a single GPU, RTX 3060 with 12GB of memory. I already tried reducing the batch size, but I still face the same error. Does this mean that I am not able to run this model with my gpu?

Let's use 1 GPUs!
Training from scratch...
The number of training data is:%d 13998
The number of test data is:%d 2874
Use Adam
Use StepLR
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6999/6999 [14:18<00:00, 8.15it/s]
Learning rate: 0.003000
Train 1, loss: 0.705842, train acc: 0.805999, train ins_iou: 0.653476
1%|█▍ | 1/90 [00:01<02:35, 1.75s/it]
Traceback (most recent call last):
File "main.py", line 442, in
train(args, io)
File "main.py", line 126, in train
test_metrics, total_per_cat_iou = test_epoch(test_loader, model, epoch, num_part, num_classes, io)
File "main.py", line 256, in test_epoch
seg_pred = model(points, norm_plt, to_categorical(label, num_classes)) # b,n,50
File "/home/goncalo/pointMLP-pytorch/env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/goncalo/pointMLP-pytorch/env/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 165, in forward
return self.module(*inputs[0], **kwargs[0])
File "/home/goncalo/pointMLP-pytorch/env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/goncalo/pointMLP-pytorch/part_segmentation/model/pointMLP.py", line 421, in forward
x = self.pre_blocks_listi # [b,d,g]
File "/home/goncalo/pointMLP-pytorch/env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/goncalo/pointMLP-pytorch/part_segmentation/model/pointMLP.py", line 254, in forward
x = self.operation(x) # [b, d, k]
File "/home/goncalo/pointMLP-pytorch/env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/goncalo/pointMLP-pytorch/env/lib/python3.7/site-packages/torch/nn/modules/container.py", line 119, in forward
input = module(input)
File "/home/goncalo/pointMLP-pytorch/env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/goncalo/pointMLP-pytorch/part_segmentation/model/pointMLP.py", line 226, in forward
return self.act(self.net2(self.net1(x)) + x)
RuntimeError: CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 12.00 GiB total capacity; 11.17 GiB already allocated; 0 bytes free; 11.23 GiB reserved in total by PyTorch)

[ma-xu]

1. you can further reduce the batch-size, which probably will have only little influence on the performance.
2. you can try multi-gpu training.