Error with GPU: CUBLAS_STATUS_EXECUTION_FAILED
skull615d opened this issue · 5 comments
Unable to run test training on GPU. Here is the error stack. (Win 10, RTX 3060Ti, CUDA 10.0 cudnn 7.6.3) Please tell me what i am doing wrong.
2023-02-05 10:33:42.061163: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll
2023-02-05 10:33:42.317036: E tensorflow/stream_executor/cuda/cuda_blas.cc:428] failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED
org.tensorflow.TensorFlowException: 2 root error(s) found.
(0) Internal: Blas GEMM launch failed : a.shape=(100, 300), b.shape=(300, 100), m=100, n=100, k=300
[[{{node MatMul_1}}]]
(1) Internal: Blas GEMM launch failed : a.shape=(100, 300), b.shape=(300, 100), m=100, n=100, k=300
[[{{node MatMul_1}}]]
[[Mean_1/_13]]
0 successful operations.
0 derived errors ignored.
Hmm, I've tested it on the following configuration . Could you miss the C++ redistributable parts?
Also, there could be a problem with the drivers for the GPU card or memory allocation https://forums.developer.nvidia.com/t/error-failed-to-run-cublas-routine-cublas-status-execution-failed/164278
matterport/Mask_RCNN#2510
Also, if you experimented with many CUDA versions, probably, you need to check twice the environment variables; the LIBRARY_PATH or PATH could contain the path to another CUDA version (it happens to me)
C++ redistributable parts and PATH is OK.
How can this density be applied in KotlinDL?
import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.Session(config=config)