How to solve CUDA out of memory ?
Opened this issue · 5 comments
I am using gtx 1660ti with 6gb vram.
Processing: 45576d11ffe4dd13222ffdd.jpeg
Speed:20.1300, 1 / 2, 0.1049
0%| | 0/800 [00:01<?, ?it/s]
Traceback (most recent call last):
File "fit_images.py", line 414, in
fit(args, device)
File "fit_images.py", line 252, in fit
rendered_img = faceverse_model.renderer.alb_renderer(mesh)
File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/pytorch3d/renderer/mesh/renderer.py", line 62, in forward
images = self.shader(fragments, meshes_world, **kwargs)
File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/pytorch3d/renderer/mesh/shader.py", line 262, in forward
colors = flat_shading(
File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/pytorch3d/renderer/mesh/shading.py", line 219, in flat_shading
ambient, diffuse, specular = _apply_lighting(
File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/pytorch3d/renderer/mesh/shading.py", line 33, in _apply_lighting
light_specular = lights.specular(
File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/pytorch3d/renderer/lighting.py", line 280, in specular
return specular(
File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/pytorch3d/renderer/lighting.py", line 152, in specular
reflect_direction = -direction + 2 * (cos_angle[..., None] * normals)
RuntimeError: CUDA out of memory. Tried to allocate 12.00 MiB (GPU 0; 5.80 GiB total capacity; 3.86 GiB already allocated; 52.88 MiB free; 3.90 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Processing: 45576d11ffe4dd13222ffdd.jpeg
Speed:5.5962, 1 / 2, 18.7387
0%| | 0/800 [00:00<?, ?it/s]
Traceback (most recent call last):
File "fit_images.py", line 414, in
fit(args, device)
File "fit_images.py", line 234, in fit
detail_img, _ = g_detail([latent_detail], input_detail * mask_detail, return_latents=True, noise=noises_detail)
File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/michael/Downloads/FaceVerse-main/network.py", line 648, in forward
out = conv1(out, latent[:, i], noise=noise1)
File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/michael/Downloads/FaceVerse-main/network.py", line 360, in forward
out = self.conv(input, style)
File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/michael/Downloads/FaceVerse-main/network.py", line 258, in forward
weight = self.scale * self.weight * style
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 5.80 GiB total capacity; 2.06 GiB already allocated; 57.94 MiB free; 2.11 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Someone please help -- Really struggling to get this working
Maybe you can try to comment out all the code related to the expression network. (Line 291 # fitting with the exp ckpt & the model loading started from Line 76)
Maybe you can try to comment out all the code related to the expression network. (Line 291 # fitting with the exp ckpt & the model loading started from Line 76)
Should I just delete fit_images line 291 to line 341 or just comment out these lines with exp ?
After deleting fit_images line 291 to line 341 and comment out line 73 to line 76
Processing: 45576d11ffe4dd13222ffdd.jpeg
Speed:5.4946, 1 / 2, 18.7387
lm: 15.3111; pho: 0.6716; noise: 0.0017; : 1%| | 7/800 [00:05<09:44, 1.36it/s
Traceback (most recent call last):
File "fit_images.py", line 365, in
fit(args, device)
File "fit_images.py", line 265, in fit
loss.backward()
File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/torch/_tensor.py", line 307, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/torch/autograd/init.py", line 154, in backward
Variable._execution_engine.run_backward(
File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/torch/autograd/function.py", line 199, in apply
return user_fn(self, *args)
File "/home/michael/Downloads/FaceVerse-main/third_libs/stylegan_ops/fused_act.py", line 80, in backward
grad_input, grad_bias = FusedLeakyReLUFunctionBackward.apply(
File "/home/michael/Downloads/FaceVerse-main/third_libs/stylegan_ops/fused_act.py", line 32, in forward
grad_input = fused.fused_bias_act(
RuntimeError: CUDA out of memory. Tried to allocate 128.00 MiB (GPU 0; 5.80 GiB total capacity; 3.54 GiB already allocated; 110.44 MiB free; 3.97 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF