GTX 2080Ti use default config out of mermory

Question

GTX 2080Ti use default config out of mermory

zhijiejia opened this issue 4 years ago · 3 comments

bash scripts/train.sh pascal 0 [0,1] 50
The first one epoch is ok, but the test phrase will out o memory

Answer 1 · 2021-03-10T13:38:27.000Z

Hi,

You could try to lower the batch_size_val (which basically represents how many tasks will be treated in parallel during evaluation). By default, I set it to 200, but this is probably too much depending on your GPU. You could set it to as low as 1 if you want it to be more memory efficient. Let me know if that works for you.

Malik

Answer 2 · 2021-03-10T13:42:06.000Z

Yes, I did it yesterday, it works well, but the val process should not be whith @torch.no_grad ? So that it won't take up the GPU memory

Answer 3 · 2021-03-10T14:25:50.000Z

Yes, you are 100% right. I dropped it somewhere along the way (as I was experimenting what would happen if you finetune the whole network --bad idea--), and never put it back. I've just added it and pushed. Thanks for noticing.