Issue while launching generate.py
DaddyWesker opened this issue · 2 comments
DaddyWesker commented
Hello.
THanks for your code. When i'm trying to launch generate file i'm getting this error:
Downloading https://vision-cair.s3.amazonaws.com/alis/lhq1024-snapshot.pkl ... done
Setting up PyTorch plugin "bias_act_plugin"... Failed!
/home/daddywesker/SingularityNet/MixingImages/alis/torch_utils/ops/bias_act.py:50: UserWarning: Failed to build CUDA kernels for bias_act. Falling back to slow reference implementation. Details:
Error building extension 'bias_act_plugin': [1/3] /usr/bin/nvcc --generate-dependencies-with-compile --dependency-output bias_act.cuda.o.d -DTORCH_EXTENSION_NAME=bias_act_plugin -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/TH -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/THC -isystem /home/daddywesker/anaconda3/envs/torch/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' --use_fast_math -std=c++14 -c /home/daddywesker/SingularityNet/MixingImages/alis/torch_utils/ops/bias_act.cu -o bias_act.cuda.o
FAILED: bias_act.cuda.o
/usr/bin/nvcc --generate-dependencies-with-compile --dependency-output bias_act.cuda.o.d -DTORCH_EXTENSION_NAME=bias_act_plugin -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/TH -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/THC -isystem /home/daddywesker/anaconda3/envs/torch/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' --use_fast_math -std=c++14 -c /home/daddywesker/SingularityNet/MixingImages/alis/torch_utils/ops/bias_act.cu -o bias_act.cuda.o
nvcc fatal : Unknown option '-generate-dependencies-with-compile'
[2/3] c++ -MMD -MF bias_act.o.d -DTORCH_EXTENSION_NAME=bias_act_plugin -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/TH -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/THC -isystem /home/daddywesker/anaconda3/envs/torch/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=1 -fPIC -std=c++14 -c /home/daddywesker/SingularityNet/MixingImages/alis/torch_utils/ops/bias_act.cpp -o bias_act.o
ninja: build stopped: subcommand failed.
warnings.warn('Failed to build CUDA kernels for bias_act. Falling back to slow reference implementation. Details:\n\n' + str(sys.exc_info()[1]))
Setting up PyTorch plugin "upfirdn2d_plugin"... Failed!
/home/daddywesker/SingularityNet/MixingImages/alis/torch_utils/ops/upfirdn2d.py:34: UserWarning: Failed to build CUDA kernels for upfirdn2d. Falling back to slow reference implementation. Details:
Error building extension 'upfirdn2d_plugin': [1/3] /usr/bin/nvcc --generate-dependencies-with-compile --dependency-output upfirdn2d.cuda.o.d -DTORCH_EXTENSION_NAME=upfirdn2d_plugin -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/TH -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/THC -isystem /home/daddywesker/anaconda3/envs/torch/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' --use_fast_math -std=c++14 -c /home/daddywesker/SingularityNet/MixingImages/alis/torch_utils/ops/upfirdn2d.cu -o upfirdn2d.cuda.o
FAILED: upfirdn2d.cuda.o
/usr/bin/nvcc --generate-dependencies-with-compile --dependency-output upfirdn2d.cuda.o.d -DTORCH_EXTENSION_NAME=upfirdn2d_plugin -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/TH -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/THC -isystem /home/daddywesker/anaconda3/envs/torch/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' --use_fast_math -std=c++14 -c /home/daddywesker/SingularityNet/MixingImages/alis/torch_utils/ops/upfirdn2d.cu -o upfirdn2d.cuda.o
nvcc fatal : Unknown option '-generate-dependencies-with-compile'
[2/3] c++ -MMD -MF upfirdn2d.o.d -DTORCH_EXTENSION_NAME=upfirdn2d_plugin -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/TH -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/THC -isystem /home/daddywesker/anaconda3/envs/torch/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=1 -fPIC -std=c++14 -c /home/daddywesker/SingularityNet/MixingImages/alis/torch_utils/ops/upfirdn2d.cpp -o upfirdn2d.o
ninja: build stopped: subcommand failed.
warnings.warn('Failed to build CUDA kernels for upfirdn2d. Falling back to slow reference implementation. Details:\n\n' + str(sys.exc_info()[1]))
Traceback (most recent call last):
File "/home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1673, in _run_ninja_build
env=env)
File "/home/daddywesker/anaconda3/envs/torch/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/daddywesker/SingularityNet/MixingImages/alis/generate.py", line 67, in <module>
img = G.synthesis(curr_ws, ws_context=curr_ws_context, left_borders_idx=curr_left_borders_idx, noise='const')
File "/home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/daddywesker/SingularityNet/MixingImages/alis/training/networks.py", line 1030, in forward
x, img = block(x, img, cur_ws, ws_context=curr_ws_context, left_borders_idx=left_borders_idx, **block_kwargs)
File "/home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/daddywesker/SingularityNet/MixingImages/alis/training/networks.py", line 903, in forward
**layer_kwargs)
File "/home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/daddywesker/SingularityNet/MixingImages/alis/training/networks.py", line 531, in forward
w_lerp_multiplier=self.cfg.patchwise.w_lerp_multiplier,
File "/home/daddywesker/SingularityNet/MixingImages/alis/torch_utils/misc.py", line 101, in decorator
return fn(*args, **kwargs)
File "/home/daddywesker/SingularityNet/MixingImages/alis/training/networks.py", line 200, in patchwise_conv2d
flip_weight=flip_weight)
File "/home/daddywesker/SingularityNet/MixingImages/alis/torch_utils/misc.py", line 101, in decorator
return fn(*args, **kwargs)
File "/home/daddywesker/SingularityNet/MixingImages/alis/training/networks.py", line 222, in patchwise_op
y = op(x, *args, **kwargs)
File "/home/daddywesker/SingularityNet/MixingImages/alis/torch_utils/misc.py", line 101, in decorator
return fn(*args, **kwargs)
File "/home/daddywesker/SingularityNet/MixingImages/alis/torch_utils/ops/conv2d_resample.py", line 139, in conv2d_resample
x = upfirdn2d.upfirdn2d(x=x, f=f, padding=[px0+pxt,px1+pxt,py0+pyt,py1+pyt], gain=up**2, flip_filter=flip_filter)
File "/home/daddywesker/SingularityNet/MixingImages/alis/torch_utils/ops/upfirdn2d.py", line 163, in upfirdn2d
if impl == 'cuda' and x.device.type == 'cuda' and _init():
File "/home/daddywesker/SingularityNet/MixingImages/alis/torch_utils/ops/upfirdn2d.py", line 32, in _init
_plugin = custom_ops.get_plugin('upfirdn2d_plugin', sources=sources, extra_cuda_cflags=['--use_fast_math'])
File "/home/daddywesker/SingularityNet/MixingImages/alis/torch_utils/custom_ops.py", line 110, in get_plugin
torch.utils.cpp_extension.load(name=module_name, verbose=verbose_build, sources=sources, **build_kwargs)
File "/home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1091, in load
keep_intermediates=keep_intermediates)
File "/home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1302, in _jit_compile
is_standalone=is_standalone)
File "/home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1407, in _write_ninja_file_and_build_library
error_prefix=f"Error building extension '{name}'")
File "/home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1683, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension 'upfirdn2d_plugin': [1/3] /usr/bin/nvcc --generate-dependencies-with-compile --dependency-output upfirdn2d.cuda.o.d -DTORCH_EXTENSION_NAME=upfirdn2d_plugin -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/TH -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/THC -isystem /home/daddywesker/anaconda3/envs/torch/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' --use_fast_math -std=c++14 -c /home/daddywesker/SingularityNet/MixingImages/alis/torch_utils/ops/upfirdn2d.cu -o upfirdn2d.cuda.o
FAILED: upfirdn2d.cuda.o
/usr/bin/nvcc --generate-dependencies-with-compile --dependency-output upfirdn2d.cuda.o.d -DTORCH_EXTENSION_NAME=upfirdn2d_plugin -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/TH -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/THC -isystem /home/daddywesker/anaconda3/envs/torch/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' --use_fast_math -std=c++14 -c /home/daddywesker/SingularityNet/MixingImages/alis/torch_utils/ops/upfirdn2d.cu -o upfirdn2d.cuda.o
nvcc fatal : Unknown option '-generate-dependencies-with-compile'
[2/3] c++ -MMD -MF upfirdn2d.o.d -DTORCH_EXTENSION_NAME=upfirdn2d_plugin -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/TH -isystem /home/daddywesker/anaconda3/envs/torch/lib/python3.7/site-packages/torch/include/THC -isystem /home/daddywesker/anaconda3/envs/torch/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=1 -fPIC -std=c++14 -c /home/daddywesker/SingularityNet/MixingImages/alis/torch_utils/ops/upfirdn2d.cpp -o upfirdn2d.o
ninja: build stopped: subcommand failed.
Process finished with exit code 1
As a clarification, generate.py is just your pynb copied to regular python script file, it's just easier for me.
Tried to search for this nvcc fatal : Unknown option '-generate-dependencies-with-compile' but currently got no clue.
I'm using pytorch 1.8, cuda 10.1, Ubuntu 20. THanks in advance for the help.
DaddyWesker commented
Well, it seems i need cuda 10.2 and higher... I'll try to install it and, if it'll help, i'll close the issue.
DaddyWesker commented
Installing cuda 11.0 and re-installing pytorch using pip solved issue for me.