Unable to run the code
Opened this issue · 0 comments
shahaamirbader commented
I am unable to run the code using conda environment. I tried running in different pytorch images I have in my server but the issue persists.
Whenever I give the following command
python3 speechgpt/src/infer/cli_infer.py --model-name-or-path "speechgpt/SpeechGPT-7B-cm" --lora-weights "speechgpt/SpeechGPT-7B-com" --s2u-dir "utils/speech2unit" --vocoder-dir "utils/vocoder" --output-dir "output"
xubuntu@794f23883d81 /data SpeechGPT speechgpt SpeechGPT 3.8.18 USAGE
❯ cd ..
xubuntu@794f23883d81 /data SpeechGPT SpeechGPT 3.8.18
❯ python3 speechgpt/src/infer/cli_infer.py --model-name-or-path "speechgpt/SpeechGPT-7B-cm" --lora-weights "speechgpt/SpeechGPT-7B-com" --s2u-dir "utils/speech2unit" --vocoder-dir "utils/vocoder" --output-dir "output"
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: /home/xubuntu/anaconda3/envs/SpeechGPT did not contain libcudart.so as expected! Searching further paths...
warn(msg)
/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/nvidia/lib64'), PosixPath('/usr/local/cuda/extras/CUPTI/lib64'), PosixPath('/usr/local/nvidia/lib'), PosixPath('/lib/i386-linux-gnu')}
warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.0
CUDA SETUP: Detected CUDA version 123
/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: Compute capability < 7.5 detected! Only slow 8-bit matmul is supported for your GPU!
warn(msg)
CUDA SETUP: Required library version not found: libbitsandbytes_cuda123_nocublaslt.so. Maybe you need to compile it from source?
CUDA SETUP: Defaulting to libbitsandbytes_cpu.so...
================================================ERROR=====================================
CUDA SETUP: CUDA detection failed! Possible reasons:
1. CUDA driver not installed
2. CUDA not installed
3. You have multiple conflicting CUDA libraries
4. Required library not pre-compiled for this bitsandbytes release!
CUDA SETUP: If you compiled from source, try again with `make CUDA_VERSION=DETECTED_CUDA_VERSION` for example, `make CUDA_VERSION=113`.
CUDA SETUP: The CUDA version for the compile might depend on your conda install. Inspect CUDA version via `conda list | grep cuda`.
================================================================================
CUDA SETUP: Something unexpected happened. Please compile from source:
git clone git@github.com:TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=123_nomatmul
python setup.py install
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.0
CUDA SETUP: Detected CUDA version 123
CUDA SETUP: Required library version not found: libbitsandbytes_cuda123_nocublaslt.so. Maybe you need to compile it from source?
CUDA SETUP: Defaulting to libbitsandbytes_cpu.so...
================================================ERROR=====================================
CUDA SETUP: CUDA detection failed! Possible reasons:
1. CUDA driver not installed
2. CUDA not installed
3. You have multiple conflicting CUDA libraries
4. Required library not pre-compiled for this bitsandbytes release!
CUDA SETUP: If you compiled from source, try again with `make CUDA_VERSION=DETECTED_CUDA_VERSION` for example, `make CUDA_VERSION=113`.
CUDA SETUP: The CUDA version for the compile might depend on your conda install. Inspect CUDA version via `conda list | grep cuda`.
================================================================================
CUDA SETUP: Something unexpected happened. Please compile from source:
git clone git@github.com:TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=123_nomatmul
python setup.py install
CUDA SETUP: Setup Failed!
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.0
CUDA SETUP: Detected CUDA version 123
CUDA SETUP: Required library version not found: libbitsandbytes_cuda123_nocublaslt.so. Maybe you need to compile it from source?
CUDA SETUP: Defaulting to libbitsandbytes_cpu.so...
================================================ERROR=====================================
CUDA SETUP: CUDA detection failed! Possible reasons:
1. CUDA driver not installed
2. CUDA not installed
3. You have multiple conflicting CUDA libraries
4. Required library not pre-compiled for this bitsandbytes release!
CUDA SETUP: If you compiled from source, try again with `make CUDA_VERSION=DETECTED_CUDA_VERSION` for example, `make CUDA_VERSION=113`.
CUDA SETUP: The CUDA version for the compile might depend on your conda install. Inspect CUDA version via `conda list | grep cuda`.
================================================================================
CUDA SETUP: Something unexpected happened. Please compile from source:
git clone git@github.com:TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=123_nomatmul
python setup.py install
CUDA SETUP: Setup Failed!
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.0
CUDA SETUP: Detected CUDA version 123
CUDA SETUP: Required library version not found: libbitsandbytes_cuda123_nocublaslt.so. Maybe you need to compile it from source?
CUDA SETUP: Defaulting to libbitsandbytes_cpu.so...
================================================ERROR=====================================
CUDA SETUP: CUDA detection failed! Possible reasons:
1. CUDA driver not installed
2. CUDA not installed
3. You have multiple conflicting CUDA libraries
4. Required library not pre-compiled for this bitsandbytes release!
CUDA SETUP: If you compiled from source, try again with `make CUDA_VERSION=DETECTED_CUDA_VERSION` for example, `make CUDA_VERSION=113`.
CUDA SETUP: The CUDA version for the compile might depend on your conda install. Inspect CUDA version via `conda list | grep cuda`.
================================================================================
CUDA SETUP: Something unexpected happened. Please compile from source:
git clone git@github.com:TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=123_nomatmul
python setup.py install
CUDA SETUP: Setup Failed!
CUDA SETUP: Something unexpected happened. Please compile from source:
git clone git@github.com:TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=123_nomatmul
python setup.py install
2024-02-24 19:31:45 | INFO | fairseq.tasks.text_to_speech | Please install tensorboardX: pip install tensorboardX
/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: /home/xubuntu/anaconda3/envs/SpeechGPT did not contain libcudart.so as expected! Searching further paths...
warn(msg)
/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/nvidia/lib64'), PosixPath('/usr/local/cuda/extras/CUPTI/lib64'), PosixPath('/usr/local/nvidia/lib'), PosixPath('/lib/i386-linux-gnu')}
warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.0
CUDA SETUP: Detected CUDA version 123
/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: Compute capability < 7.5 detected! Only slow 8-bit matmul is supported for your GPU!
warn(msg)
CUDA SETUP: Required library version not found: libbitsandbytes_cuda123_nocublaslt.so. Maybe you need to compile it from source?
CUDA SETUP: Defaulting to libbitsandbytes_cpu.so...
================================================ERROR=====================================
CUDA SETUP: CUDA detection failed! Possible reasons:
1. CUDA driver not installed
2. CUDA not installed
3. You have multiple conflicting CUDA libraries
4. Required library not pre-compiled for this bitsandbytes release!
CUDA SETUP: If you compiled from source, try again with `make CUDA_VERSION=DETECTED_CUDA_VERSION` for example, `make CUDA_VERSION=113`.
CUDA SETUP: The CUDA version for the compile might depend on your conda install. Inspect CUDA version via `conda list | grep cuda`.
================================================================================
CUDA SETUP: Something unexpected happened. Please compile from source:
git clone git@github.com:TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=123_nomatmul
python setup.py install
CUDA SETUP: Setup Failed!
CUDA SETUP: Something unexpected happened. Please compile from source:
git clone git@github.com:TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=123_nomatmul
python setup.py install
CUDA SETUP: Something unexpected happened. Please compile from source:
git clone git@github.com:TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=123_nomatmul
python setup.py install
Traceback (most recent call last):
File "/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/site-packages/transformers/utils/import_utils.py", line 1184, in _get_module
return importlib.import_module("." + module_name, self.__name__)
File "/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 843, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/site-packages/transformers/generation/utils.py", line 27, in <module>
from ..integrations.deepspeed import is_deepspeed_zero3_enabled
File "/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/site-packages/transformers/integrations/__init__.py", line 14, in <module>
from .bitsandbytes import (
File "/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/site-packages/transformers/integrations/bitsandbytes.py", line 11, in <module>
import bitsandbytes as bnb
File "/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/site-packages/bitsandbytes/__init__.py", line 7, in <module>
from .autograd._functions import (
File "/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/site-packages/bitsandbytes/autograd/__init__.py", line 1, in <module>
from ._functions import undo_layout, get_inverse_transform_indices
File "/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/site-packages/bitsandbytes/autograd/_functions.py", line 9, in <module>
import bitsandbytes.functional as F
File "/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/site-packages/bitsandbytes/functional.py", line 17, in <module>
from .cextension import COMPILED_WITH_CUDA, lib
File "/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/site-packages/bitsandbytes/cextension.py", line 22, in <module>
raise RuntimeError('''
RuntimeError:
CUDA Setup failed despite GPU being available. Inspect the CUDA SETUP outputs above to fix your environment!
If you cannot find any issues and suspect a bug, please open an issue with detals about your environment:
https://github.com/TimDettmers/bitsandbytes/issues
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/site-packages/transformers/utils/import_utils.py", line 1184, in _get_module
return importlib.import_module("." + module_name, self.__name__)
File "/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 843, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/site-packages/transformers/modeling_utils.py", line 39, in <module>
from .generation import GenerationConfig, GenerationMixin
File "<frozen importlib._bootstrap>", line 1039, in _handle_fromlist
File "/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/site-packages/transformers/utils/import_utils.py", line 1174, in __getattr__
module = self._get_module(self._class_to_module[name])
File "/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/site-packages/transformers/utils/import_utils.py", line 1186, in _get_module
raise RuntimeError(
RuntimeError: Failed to import transformers.generation.utils because of the following error (look up to see its traceback):
CUDA Setup failed despite GPU being available. Inspect the CUDA SETUP outputs above to fix your environment!
If you cannot find any issues and suspect a bug, please open an issue with detals about your environment:
https://github.com/TimDettmers/bitsandbytes/issues
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "speechgpt/src/infer/cli_infer.py", line 13, in <module>
from peft import PeftModel
File "/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/site-packages/peft/__init__.py", line 22, in <module>
from .mapping import MODEL_TYPE_TO_PEFT_MODEL_MAPPING, PEFT_TYPE_TO_CONFIG_MAPPING, get_peft_config, get_peft_model
File "/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/site-packages/peft/mapping.py", line 16, in <module>
from .peft_model import (
File "/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/site-packages/peft/peft_model.py", line 27, in <module>
from transformers import PreTrainedModel
File "<frozen importlib._bootstrap>", line 1039, in _handle_fromlist
File "/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/site-packages/transformers/utils/import_utils.py", line 1174, in __getattr__
module = self._get_module(self._class_to_module[name])
File "/home/xubuntu/anaconda3/envs/SpeechGPT/lib/python3.8/site-packages/transformers/utils/import_utils.py", line 1186, in _get_module
raise RuntimeError(
RuntimeError: Failed to import transformers.modeling_utils because of the following error (look up to see its traceback):
Failed to import transformers.generation.utils because of the following error (look up to see its traceback):
CUDA Setup failed despite GPU being available. Inspect the CUDA SETUP outputs above to fix your environment!
If you cannot find any issues and suspect a bug, please open an issue with detals about your environment:
https://github.com/TimDettmers/bitsandbytes/issues```
If I run nvidia-smi and nvcc --version commands to check my cuda versions they are compatible.
Sat Feb 24 19:32:17 2024
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.147.05 Driver Version: 525.147.05 CUDA Version: 12.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla V100-DGXS... Off | 00000000:07:00.0 Off | 0 |
| N/A 37C P0 37W / 300W | 185MiB / 32768MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 Tesla V100-DGXS... Off | 00000000:08:00.0 Off | 0 |
| N/A 36C P0 39W / 300W | 9MiB / 32768MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 2 Tesla V100-DGXS... Off | 00000000:0E:00.0 Off | 0 |
| N/A 37C P0 39W / 300W | 9MiB / 32768MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 3 Tesla V100-DGXS... Off | 00000000:0F:00.0 Off | 0 |
| N/A 35C P0 37W / 300W | 9MiB / 32768MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
+-----------------------------------------------------------------------------+
❯ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Wed_Nov_22_10:17:15_PST_2023
Cuda compilation tools, release 12.3, V12.3.107
Build cuda_12.3.r12.3/compiler.33567101_0