RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED
wzgpeter opened this issue · 0 comments
A100 platform
cuda118
run command: python gradio_canny2image.py
log show as below:
ad.
result_gallery = gr.Gallery(label='Output', show_label=False, elem_id="gallery").style(grid=2, height='auto')
gradio_canny2image.py:98: GradioDeprecationWarning: The 'grid' parameter will be deprecated. Please use 'grid_cols' in the constructor instead.
result_gallery = gr.Gallery(label='Output', show_label=False, elem_id="gallery").style(grid=2, height='auto')
Running on local URL: http://0.0.0.0:5000
IMPORTANT: You are using gradio version 3.38.0, however version 4.29.0 is available, please upgrade.
Running on public URL: https://49b5cf058a69109bd7.gradio.live
This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run gradio deploy from Terminal to deploy to Spaces (https://huggingface.co/spaces)
curr-time: 0:00:00.001951
Global seed set to 1706203096
Data shape for DDIM sampling is (1, 4, 104, 64), eta 0.0
Running DDIM Sampling with 20 timesteps
DDIM Sampler: 0%| | 0/20 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/gradio/routes.py", line 442, in run_predict
output = await app.get_blocks().process_api(
File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/gradio/blocks.py", line 1389, in process_api
result = await self.call_function(
File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/gradio/blocks.py", line 1094, in call_function
prediction = await anyio.to_thread.run_sync(
File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 2357, in run_sync_in_worker_thread
return await future
File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 864, in run
result = context.run(func, *args)
File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/gradio/utils.py", line 703, in wrapper
response = f(*args, **kwargs)
File "gradio_canny2image.py", line 59, in process
samples, intermediates = ddim_sampler.sample(ddim_steps, num_samples,
File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/home/ControlNet/cldm/ddim_hacked.py", line 103, in sample
samples, intermediates = self.ddim_sampling(conditioning, size,
File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/home/ControlNet/cldm/ddim_hacked.py", line 163, in ddim_sampling
outs = self.p_sample_ddim(img, cond, ts, index=index, use_original_steps=ddim_use_original_steps,
File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/home/ControlNet/cldm/ddim_hacked.py", line 190, in p_sample_ddim
model_t = self.model.apply_model(x, t, c)
File "/home/ControlNet/cldm/cldm.py", line 337, in apply_model
control = self.control_model(x=x_noisy, hint=torch.cat(cond['c_concat'], 1), timesteps=t, context=cond_txt)
File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ControlNet/cldm/cldm.py", line 288, in forward
guided_hint = self.input_hint_block(hint, emb, context)
File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ControlNet/ldm/modules/diffusionmodules/openaimodel.py", line 86, in forward
x = layer(x)
File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 458, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/opt/conda/envs/controlnet/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 454, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED
pip list show as below:
Package Version
absl-py 2.1.0
aiofiles 23.2.1
aiohappyeyeballs 2.4.3
aiohttp 3.10.9
aiosignal 1.3.1
altair 5.4.1
annotated-types 0.7.0
antlr4-python3-runtime 4.8
anyio 4.5.0
async-timeout 4.0.3
attrs 24.2.0
cachetools 5.5.0
certifi 2024.8.30
charset-normalizer 3.4.0
click 8.1.7
contourpy 1.1.1
cycler 0.12.1
diffusers 0.30.3
einops 0.3.0
exceptiongroup 1.2.2
fastapi 0.115.0
ffmpy 0.4.0
filelock 3.16.1
fonttools 4.54.1
frozenlist 1.4.1
fsspec 2024.9.0
ftfy 6.2.3
future 1.0.0
google-auth 2.35.0
google-auth-oauthlib 1.0.0
gradio 3.38.0
gradio_client 1.3.0
grpcio 1.66.2
h11 0.14.0
httpcore 1.0.6
httpx 0.27.2
huggingface-hub 0.25.2
idna 3.10
importlib_metadata 8.5.0
importlib_resources 6.4.5
Jinja2 3.1.4
jsonschema 4.23.0
jsonschema-specifications 2023.12.1
kiwisolver 1.4.7
lightning-utilities 0.11.7
linkify-it-py 2.0.3
Markdown 3.7
markdown-it-py 2.2.0
MarkupSafe 2.1.5
matplotlib 3.7.5
mdit-py-plugins 0.3.3
mdurl 0.1.2
mpmath 1.3.0
multidict 6.1.0
narwhals 1.9.2
networkx 3.1
numpy 1.24.4
nvidia-cublas-cu11 11.11.3.6
nvidia-cublas-cu12 12.1.3.1
nvidia-cuda-cupti-cu11 11.8.87
nvidia-cuda-cupti-cu12 12.1.105
nvidia-cuda-nvrtc-cu11 11.8.89
nvidia-cuda-nvrtc-cu12 12.1.105
nvidia-cuda-runtime-cu11 11.8.89
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu11 9.1.0.70
nvidia-cudnn-cu12 9.1.0.70
nvidia-cufft-cu11 10.9.0.58
nvidia-cufft-cu12 11.0.2.54
nvidia-curand-cu11 10.3.0.86
nvidia-curand-cu12 10.3.2.106
nvidia-cusolver-cu11 11.4.1.48
nvidia-cusolver-cu12 11.4.5.107
nvidia-cusparse-cu11 11.7.5.86
nvidia-cusparse-cu12 12.1.0.106
nvidia-nccl-cu11 2.20.5
nvidia-nccl-cu12 2.20.5
nvidia-nvjitlink-cu12 12.6.77
nvidia-nvtx-cu11 11.8.86
nvidia-nvtx-cu12 12.1.105
oauthlib 3.2.2
omegaconf 2.1.1
open-clip-torch 2.0.2
opencv-python 4.10.0.84
orjson 3.10.7
packaging 24.1
pandas 2.0.3
pillow 10.4.0
pip 24.2
pkgutil_resolve_name 1.3.10
propcache 0.2.0
protobuf 5.28.2
pyasn1 0.6.1
pyasn1_modules 0.4.1
pycryptodome 3.21.0
pydantic 2.9.2
pydantic_core 2.23.4
pyDeprecate 0.3.1
pydub 0.25.1
Pygments 2.18.0
pyparsing 3.1.4
python-dateutil 2.9.0.post0
python-multipart 0.0.12
pytorch-lightning 1.5.0
pytz 2024.2
PyYAML 6.0.2
referencing 0.35.1
regex 2024.9.11
requests 2.32.3
requests-oauthlib 2.0.0
rich 13.9.2
rpds-py 0.20.0
rsa 4.9
ruff 0.6.9
safetensors 0.4.5
semantic-version 2.10.0
setuptools 75.1.0
shellingham 1.5.4
six 1.16.0
sniffio 1.3.1
starlette 0.38.6
sympy 1.13.3
tensorboard 2.14.0
tensorboard-data-server 0.7.2
timm 1.0.9
tokenizers 0.12.1
tomlkit 0.12.0
torch 2.4.1+cu118
torchaudio 2.4.1+cu118
torchmetrics 1.4.2
torchvision 0.19.1+cu118
tqdm 4.66.5
transformers 4.19.2
triton 3.0.0
typer 0.12.5
typing_extensions 4.12.2
tzdata 2024.2
uc-micro-py 1.0.3
urllib3 2.2.3
uvicorn 0.31.1
wcwidth 0.2.13
websockets 11.0.3
Werkzeug 3.0.4
wheel 0.44.0
xformers 0.0.28.post1
yarl 1.14.0
zipp 3.20.2