CUDA out of memory even with 8 GPUs
Opened this issue · 6 comments
Hi, I have tried to train the model with your optimize_nha.py,
but CUDA out of memory came out sometimes in the beginning of training or after the 150 epoch.
I used gpus=8 in .ini file.
I have tried to reduce batch size but it didn't work. e.g. train_batch_size = [1, 1, 1]
The GPUs are Tesla V100-SXM2 16G.
Could you let me know what is the problem?
I think it is enough to train the code.
You may change the texture mlp's model size to run, like 256 to 128. But I am not clear about the effect on the final result.
@jkhong99 is it solved? I am also facing the same issue although I changed the texture mlp model size
I used 3layers and 4 layers instead 6 and 8 layers. It was available.
I have not tried to change texture mlp dimension yet.
Worked on my system as well after reducing the layers. But changing only texture mlp dimension doesn't work
Hi, What type of machine do I need to obtain the sufficient results demonstrated in the paper? At least A100/V100? What if 1080Ti or 2080Ti?
Hi, What type of machine do I need to obtain the sufficient results demonstrated in the paper? At least A100/V100? What if 1080Ti or 2080Ti?
No, It's not enough, you should use 3090TI or maybe better.