rstrivedi/Melting-Pot-Contest-2023

TensorFlow GPU Detection Issue in WSL2

Closed this issue · 3 comments

itstyren commented

Description:

I have followed the installation guidelines for setting up TensorFlow in a WSL2 environment, but I'm encountering an issue where TensorFlow is unable to detect my GPU. I would like assistance in resolving this issue.

Steps to Reproduce:

The following command python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))" return

2023-09-04 22:53:51.646492: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-09-04 22:53:51.723849: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib/wsl/lib::/lib:/home/tyren/miniconda3/lib/
2023-09-04 22:53:51.723880: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2023-09-04 22:53:52.212623: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib/wsl/lib::/lib:/home/tyren/miniconda3/lib/
2023-09-04 22:53:52.212707: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib/wsl/lib::/lib:/home/tyren/miniconda3/lib/
2023-09-04 22:53:52.212726: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2023-09-04 22:53:53.303714: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib/wsl/lib::/lib:/home/tyren/miniconda3/lib/
2023-09-04 22:53:53.303816: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublas.so.11'; dlerror: libcublas.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib/wsl/lib::/lib:/home/tyren/miniconda3/lib/
2023-09-04 22:53:53.303870: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublasLt.so.11'; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib/wsl/lib::/lib:/home/tyren/miniconda3/lib/
2023-09-04 22:53:53.303919: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib/wsl/lib::/lib:/home/tyren/miniconda3/lib/
2023-09-04 22:53:53.400425: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusparse.so.11'; dlerror: libcusparse.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib/wsl/lib::/lib:/home/tyren/miniconda3/lib/
2023-09-04 22:53:53.400533: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib/wsl/lib::/lib:/home/tyren/miniconda3/lib/
2023-09-04 22:53:53.400558: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1934] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[]

While the torch code can reports that GPU device is available:

import torch

if torch.cuda.is_available():
    num_gpus = torch.cuda.device_count()
    print(f"PyTorch can use {num_gpus} GPU(s).")
    for i in range(num_gpus):
        gpu_properties = torch.cuda.get_device_properties(i)
        print(f"GPU {i}: {gpu_properties.name}, Memory: {gpu_properties.total_memory / (1024**3):.2f}GB")
else:
    print("PyTorch cannot use GPU. Running on CPU.")

# PyTorch can use 1 GPU(s).
# GPU 0: NVIDIA RTX A1000 Laptop GPU, Memory: 4.00GB

Note:

I have already checked the following:

Any assistance in resolving this issue would be greatly appreciated.

Thank you for your help!

rstrivedi commented

Thanks for your question. I have not used WSL before but I have got similar error in my setup before. It is most likely because TF is not able to find libcudart.so.11.0 library based on your LD_LIBRARY_PATH. This is also known issue when using conda environment.

One thing to try is installing cuda toolkit/cudnn for conda: https://anaconda.org/anaconda/cudatoolkit
And then it is important to make sure that library path points to where the libcudart.so.11.0 is located (either in cuda-xx folder if you have system wide cuda) or the lib folder inside your conda environment hen you install for conda as I mentioned above.

Hope this helps.

itstyren commented

Thank you very much for your quick reply. Your solution has resolved my issue. For anyone else facing a similar problem, you can also find additional information at this link.

saidineshpola commented

For anyone facing this error on conda env,
fix LD_LIBRARY_PATH by running this.

CUDNN_PATH=$(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))
export LD_LIBRARY_PATH=$CUDNN_PATH/lib:$CONDA_PREFIX/lib/:$LD_LIBRARY_PATH