TomographicImaging/CCPi-Regularisation-Toolkit

Cuda PTX Error with RTX3090 on windows.

gfardell opened this issue · 2 comments

gfardell commented

Trying to run v24.0.1 (installed from anaconda) via CIL I get an error:

Error: C:\Users\ofn77899\Dev\CCPi-Regularisation-Toolkit\src\Core\regularisers_GPU\TV_FGP_GPU_core.cu:408, code: 222, reason: the provided PTX was compiled with an unsupported toolchain.

This is on a windows, with an RTX3090 and driver version 546.01

@paskino how did you build the binaries? Which version of CUDA Toolkit did you use?

paskino commented

On my laptop I have CUDA 12.4, which might not work for you?

gfardell commented

This probably needs splitting in to multiple issues, but it's useful to have the discussion in one place.

The drivers for 12.4 were released in March and I don't update that often - so that is the route problem. But there are multiple issues here.

  • First we should compile with an older version of cuda toolkit. Video drivers are backward compatible so this means every more recent driver than the supported version will run it.

  • The error is caught by the cuda code and it writes to stderr but not handled as far as I can see the return value is not checked and the execution continues.

CCPi-Regularisation-Toolkit/src/Core/regularisers_GPU/TV_FGP_GPU_core.cu

Line 524 in 3510f64

checkCudaErrors(cudaPeekAtLastError() );

CCPi-Regularisation-Toolkit/src/Core/regularisers_GPU/shared.h

Lines 28 to 38 in 3510f64

#define checkCudaErrors(call) \
{ \
const cudaError_t error = call; \
if (error != cudaSuccess) \
{ \
fprintf(stderr, "Error: %s:%d, ", __FILE__, __LINE__); \
fprintf(stderr, "code: %d, reason: %s\n", error, \
cudaGetErrorString(error)); \
return -1; \
} \
}

  • From the CIL side the error passes almost silently, running through the terminal you see the stdout but we don't capture/watch it. I've included the CIL code and output below.

code:

from cil.plugins.ccpi_regularisation.functions import FGP_TV
from cil.utilities.display import show2D
from cil.utilities import dataexample


data = dataexample.SIMULATED_SPHERE_VOLUME.get()
show2D(data)


TV_cpu = FGP_TV(max_iteration=10, device = 'cpu')
result_cpu = TV_cpu.proximal(data, tau=1.0)

TV_gpu = FGP_TV(max_iteration=10, device = 'gpu')
result_gpu = TV_gpu.proximal(data, tau=1.0)

show2D([result_cpu, result_gpu], title = ['CPU', 'GPU'])

output:

C:\Users\gemst\GitHub\work-CIL\reg_tlk_tests.py:11: UserWarning: Note that the default behaviour now sets the nonnegativity constraint to False 
  TV_cpu = FGP_TV(max_iteration=10, device = 'cpu')
C:\Users\gemst\GitHub\work-CIL\reg_tlk_tests.py:14: UserWarning: Note that the default behaviour now sets the nonnegativity constraint to False 
  TV_cpu = FGP_TV(max_iteration=10, device = 'gpu')
Error: C:\Users\ofn77899\Dev\CCPi-Regularisation-Toolkit\src\Core\regularisers_GPU\TV_FGP_GPU_core.cu:524, code: 222, reason: the provided PTX was compiled with an unsupported toolchain.

Image