GPU Scalability Bug (GPU 0 has 4x the vram of all others)
Skylion007 opened this issue · 2 comments
Skylion007 commented
I tried running this on multiple GPUs and noticed all the processes allocate memory on GPU number 1. This essentially limits me to a batch size of just 1 when each GPU can support closer to a batch size of 4 on their own.
rosinality commented
You can try to reduce training.n_sample in https://github.com/rosinality/alias-free-gan-pytorch/blob/main/config/config-t.jsonnet#L34.
Skylion007 commented
Confirmed, the difference is entirely caused by the sampling code, I optimized it and submitted a PR.