CUDA out of memory
Closed this issue · 4 comments
yezifeiafei commented
I received a 'CUDA out of memory' error while running test code.
How much GPU memory is required to run the code?
If I want to run it on a 32GB V100, what should I do?
shanemankiw commented
a 32GB A100 should be more than enough. are you running rotation estimation or panorama generation?
I received a 'CUDA out of memory' error while running test code. How much GPU memory is required to run the code? If I want to run it on a 32GB V100, what should I do?
yezifeiafei commented
panorama generation.
I just ran python public_test_single_w_prompt.py.
Here is the log message
logging improved.
No module 'xformers'. Proceeding without it.
No_Rota_Inpaint: Running in eps-prediction mode
DiffusionWrapper has 859.54 M params.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
Loaded model config from [models/norota_inpaint.yaml]
Loaded state_dict from [pretrained_models/norota_clean.ckpt]
Data shape for DDIM sampling is (1, 4, 64, 128), eta 0.0
Running DDIM Sampling with 50 timesteps
DDIM Sampler: 0%| | 0/50 [00:00<?, ?it/s]
Traceback (most recent call last):
File "public_test_single_w_prompt.py", line 102, in <module>
image_callback.log_img(model, batch, batch_idx=b_idx, split="test", flip=False)
File "/code_dir/Panodiff/cldm/logger.py", line 58, in log_img
images = pl_module.log_images(batch, split=split, **self.log_images_kwargs)
File "/root/miniconda3/envs/pano/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/code_dir/Panodiff/cldm/cldm.py", line 1942, in log_images
samples_cfg, _ = self.sample_log(
File "/root/miniconda3/envs/pano/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/code_dir/Panodiff/cldm/cldm.py", line 1820, in sample_log
samples, intermediates = ddim_sampler.sample(ddim_steps, batch_size, shape, cond, verbose=False, **kwargs)
File "/root/miniconda3/envs/pano/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/code_dir/Panodiff/ldm/models/diffusion/ddim.py", line 104, in sample
samples, intermediates = self.ddim_sampling(conditioning, size,
File "/root/miniconda3/envs/pano/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/code_dir/Panodiff/ldm/models/diffusion/ddim.py", line 748, in ddim_sampling
outs = self.p_sample_ddim(img, cond, ts, index=index, use_original_steps=ddim_use_original_steps,
File "/root/miniconda3/envs/pano/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/code_dir/Panodiff/ldm/models/diffusion/ddim.py", line 809, in p_sample_ddim
model_uncond, model_t = self.model.apply_model(x_in, t_in, c_in).chunk(2)
File "/code_dir/Panodiff/cldm/cldm.py", line 1808, in apply_model
control = self.control_model(x=xc, hint=torch.cat(cond['control_input'], 1), timesteps=t, context=cond_txt)
File "/root/miniconda3/envs/pano/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/code_dir/Panodiff/cldm/cldm.py", line 309, in forward
h = module(h, emb, context)
File "/root/miniconda3/envs/pano/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/code_dir/Panodiff/ldm/modules/diffusionmodules/openaimodel.py", line 84, in forward
x = layer(x, context)
File "/root/miniconda3/envs/pano/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/code_dir/Panodiff/ldm/modules/attention.py", line 334, in forward
x = block(x, context=context[i])
File "/root/miniconda3/envs/pano/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/code_dir/Panodiff/ldm/modules/attention.py", line 269, in forward
return checkpoint(self._forward, (x, context), self.parameters(), self.checkpoint)
File "/code_dir/Panodiff/ldm/modules/diffusionmodules/util.py", line 114, in checkpoint
return CheckpointFunction.apply(func, len(inputs), *args)
File "/root/miniconda3/envs/pano/lib/python3.8/site-packages/torch/autograd/function.py", line 506, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "/code_dir/Panodiff/ldm/modules/diffusionmodules/util.py", line 129, in forward
output_tensors = ctx.run_function(*ctx.input_tensors)
File "/code_dir/Panodiff/ldm/modules/attention.py", line 272, in _forward
x = self.attn1(self.norm1(x), context=context if self.disable_self_attn else None) + x
File "/root/miniconda3/envs/pano/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/code_dir/Panodiff/ldm/modules/attention.py", line 177, in forward
sim = einsum('b i d, b j d -> b i j', q, k) * self.scale
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 9.00 GiB (GPU 0; 31.75 GiB total capacity; 14.73 GiB already allocated; 7.55 GiB free; 23.82 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
shanemankiw commented
thats strange. could you try install xformers?
yezifeiafei commented
thats strange. could you try install xformers?
It works! Thanks for your reply.