Trying to run on GPU?
Closed this issue · 2 comments
LungTumorMask trying to run on GPU causing CUDA error on Ubuntu20.04. Solution: Force to run on CPU?
NVIDIA-SMI 535.86.05 Driver Version: 535.86.05 CUDA Version: 12.2
Terminal log:
[11:47:40.568][0] Command to run: ../../medtekAI/medtekAI/Docker-DeepSintef/venvLungTumorMask/bin/python -u /home/fraxinus/Fraxinus/Fraxinus_2023.09.04-dev+develop.0bc0ab_Linux-5.15.0-75-generic/Fraxinus/config/profiles/Laboratory/filter_scripts/./scripts/python_tumor_segmentation/segmentTumors.py /home/fraxinus/Patients/Planning/2023-09-22_11-Test.cx3/Images/THX_2_20230922T114732.mhd /home/fraxinus/Patients/Planning/2023-09-22_11-Test.cx3/Images/THX_2_20230922T114732_tumors.mhd
[11:47:40.568][0] Launching ScriptFilter: [../../medtekAI/medtekAI/Docker-DeepSintef/venvLungTumorMask/bin/python -u /home/fraxinus/Fraxinus/Fraxinus_2023.09.04-dev+develop.0bc0ab_Linux-5.15.0-75-generic/Fraxinus/config/profiles/Laboratory/filter_scripts/./scripts/python_tumor_segmentation/segmentTumors.py /home/fraxinus/Patients/Planning/2023-09-22_11-Test.cx3/Images/THX_2_20230922T114732.mhd /home/fraxinus/Patients/Planning/2023-09-22_11-Test.cx3/Images/THX_2_20230922T114732_tumors.mhd ]
[11:47:40.581][3] GenericScriptFilter process starting
[11:47:40.669][0] Input file: /home/fraxinus/Patients/Planning/2023-09-22_11-Test.cx3/Images/THX_2_20230922T114732.mhd
Output file: /home/fraxinus/Patients/Planning/2023-09-22_11-Test.cx3/Images/THX_2_20230922T114732_tumors.mhd
Arguments:
[11:48:00.650][0] Still executing Script, [0:20] ...
[11:48:00.713][0] Loading model...
Preprocessing image...
Segmenting lungs...
0%| | 0/147 [00:00<?, ?it/s]
0%| | 0/147 [00:00<?, ?it/s]
[11:48:00.766][0]
[11:48:00.766][0] Traceback (most recent call last):
File "/home/fraxinus/Fraxinus/Fraxinus_2023.09.04-dev+develop.0bc0ab_Linux-5.15.0-75-generic/Fraxinus/../../medtekAI/medtekAI/Docker-DeepSintef/venvLungTumorMask/bin/lungtumormask", line 8, in
sys.exit(main())
File "/home/fraxinus/Fraxinus/medtekAI/medtekAI/Docker-DeepSintef/venvLungTumorMask/lib/python3.8/site-packages/lungtumormask/main.py", line 29, in main
mask.mask(args.input, args.output, args.lung_filter, args.threshold, args.radius, args.batch_size)
File "/home/fraxinus/Fraxinus/medtekAI/medtekAI/Docker-DeepSintef/venvLungTumorMask/lib/python3.8/site-packages/lungtumormask/mask.py", line 23, in mask
preprocess_dump = preprocess(image_path, batch_size)
File "/home/fraxinus/Fraxinus/medtekAI/medtekAI/Docker-DeepSintef/venvLungTumorMask/lib/python3.8/site-packages/lungtumormask/dataprocessing.py", line 142, in preprocess
masked_lungs = mask_lung(image_path, batch_size=batch_size)
File "/home/fraxinus/Fraxinus/medtekAI/medtekAI/Docker-DeepSintef/venvLungTumorMask/lib/python3.8/site-packages/lungtumormask/dataprocessing.py", line 48, in mask_lung
prediction = model(X)
File "/home/fraxinus/Fraxinus/medtekAI/medtekAI/Docker-DeepSintef/venvLungTumorMask/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/home/fraxinus/Fraxinus/medtekAI/medtekAI/Docker-DeepSintef/venvLungTumorMask/lib/python3.8/site-packages/lungmask/resunet.py", line 61, in forward
x = down(x)
File "/home/fraxinus/Fraxinus/medtekAI/medtekAI/Docker-DeepSintef/venvLungTumorMask/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/home/fraxinus/Fraxinus/medtekAI/medtekAI/Docker-DeepSintef/venvLungTumorMask/lib/python3.8/site-packages/lungmask/resunet.py", line 109, in forward
out = self.block(x)
File "/home/fraxinus/Fraxinus/medtekAI/medtekAI/Docker-DeepSintef/venvLungTumorMask/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/home/fraxinus/Fraxinus/medtekAI/medtekAI/Docker-DeepSintef/venvLungTumorMask/lib/python3.8/site-packages/torch/nn/modules/container.py", line 141, in forward
input = module(input)
File "/home/fraxinus/Fraxinus/medtekAI/medtekAI/Docker-DeepSintef/venvLungTumorMask/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/home/fraxinus/Fraxinus/medtekAI/medtekAI/Docker-DeepSintef/venvLungTumorMask/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 447, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/home/fraxinus/Fraxinus/medtekAI/medtekAI/Docker-DeepSintef/venvLungTumorMask/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 443, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
[11:48:00.901][0] Traceback (most recent call last):
File "/home/fraxinus/Fraxinus/Fraxinus_2023.09.04-dev+develop.0bc0ab_Linux-5.15.0-75-generic/Fraxinus/config/profiles/Laboratory/filter_scripts/./scripts/python_tumor_segmentation/segmentTumors.py", line 82, in
main(sys.argv[1:])
File "/home/fraxinus/Fraxinus/Fraxinus_2023.09.04-dev+develop.0bc0ab_Linux-5.15.0-75-generic/Fraxinus/config/profiles/Laboratory/filter_scripts/./scripts/python_tumor_segmentation/segmentTumors.py", line 78, in main
segmentTumors(input_image_path, output_image_path)
File "/home/fraxinus/Fraxinus/Fraxinus_2023.09.04-dev+develop.0bc0ab_Linux-5.15.0-75-generic/Fraxinus/config/profiles/Laboratory/filter_scripts/./scripts/python_tumor_segmentation/segmentTumors.py", line 16, in segmentTumors
sitk.WriteImage(sitk.ReadImage(filenameOutput_nii_gz), filenameOutput)
File "/home/fraxinus/Fraxinus/medtekAI/medtekAI/Docker-DeepSintef/venvLungTumorMask/lib/python3.8/site-packages/SimpleITK/extra.py", line 375, in ReadImage
return reader.Execute()
File "/home/fraxinus/Fraxinus/medtekAI/medtekAI/Docker-DeepSintef/venvLungTumorMask/lib/python3.8/site-packages/SimpleITK/SimpleITK.py", line 8430, in Execute
[11:48:00.901][0]
[11:48:00.901][0]
[11:48:00.904][0] return _SimpleITK.ImageFileReader_Execute(self)
RuntimeError: Exception thrown in SimpleITK ImageFileReader_Execute: /tmp/SimpleITK/Code/IO/src/sitkImageReaderBase.cxx:97:
sitk::ERROR: The file "/home/fraxinus/Patients/Planning/2023-09-22_11-Test.cx3/Images/THX_2_20230922T114732_tumors.nii.gz" does not exist.
[11:48:00.916][3] disconnectProcess
[11:48:00.916][3] disconnecting
[11:48:00.916][3] deleteProcess
[11:48:00.916][3] deleting
[11:48:00.917][6] Done "Script": [20.35s]
Fixed in d77cfd4.
New release is out: https://github.com/VemundFredriksen/LungTumorMask/releases/tag/v1.3.0
You can now disable the GPU by simply appending --cpu to the CLI run command.