Inference를 돌리다가 이런 오류가 떴네요.

Question

Inference를 돌리다가 이런 오류가 떴네요.

Bleking opened this issue 7 months ago · 1 comments

안녕하세요. LLaVA를 이용해서 석사 논문 준비중인 대학원생입니다. 파인튜닝할 수 있는 멀티모달 LLM을 찾던 도중 이 모델을 알게되어 사용하고 있었습니다.

경로 설정해놓고 아나콘다 가상환경도 만든 다음에 세팅하라는대로 다 했고요, 적혀있던 inference 명령어 그대로 실행했는데, 이런 오류가 떴습니다. CUDA 관련 오류인걸로 보이네요.

/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:106: UserWarning:

================================================================================
WARNING: Manual override via BNB_CUDA_VERSION env variable detected!
BNB_CUDA_VERSION=XXX can be used to load a bitsandbytes version that is different from the PyTorch CUDA version.
If this was unintended set the BNB_CUDA_VERSION variable to an empty string: export BNB_CUDA_VERSION=
If you use the manual override make sure the right libcudart.so is in your LD_LIBRARY_PATH
For example by adding the following to your .bashrc: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<path_to_cuda_dir/lib64
Loading CUDA version: BNB_CUDA_VERSION=123
================================================================================

warn((f'\n\n{"="*80}\n'
False

===================================BUG REPORT===================================
/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:166: UserWarning: Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

warn(msg)

/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:166: UserWarning: /home/work/anaconda3/envs/kollava did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
warn(msg)
The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/cuda/extras/CUPTI/lib64'), PosixPath('/usr/local/nvidia/lib64')}
/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:166: UserWarning: /usr/local/cuda/compat/lib.real:/usr/local/lib/python3.10/dist-packages/torch/lib:/usr/local/lib/python3.10/dist-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda-12.3:/usr/local/cuda-12.3/include:/usr/include/x86_64-linux-gnu did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
warn(msg)
The following directories listed in your path were found to be non-existent: {PosixPath('7080/bai/ngc-pytorch'), PosixPath('bai-repo'), PosixPath('24.01-pytorch2.2-py310-cuda12.3')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
DEBUG: Possible options found for libcudart.so: {PosixPath('/usr/local/cuda/lib64/libcudart.so')}
CUDA SETUP: PyTorch settings found: CUDA_VERSION=117, Highest Compute Capability: 7.5.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Required library version not found: libbitsandbytes_cuda117.so. Maybe you need to compile it from source?
CUDA SETUP: Defaulting to libbitsandbytes_cpu.so...

================================================ERROR=====================================
CUDA SETUP: CUDA detection failed! Possible reasons:

You need to manually override the PyTorch CUDA version. Please see: "https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA driver not installed
CUDA not installed
You have multiple conflicting CUDA libraries
Required library not pre-compiled for this bitsandbytes release!
CUDA SETUP: If you compiled from source, try again with make CUDA_VERSION=DETECTED_CUDA_VERSION for example, make CUDA_VERSION=113.
CUDA SETUP: The CUDA version for the compile might depend on your conda install. Inspect CUDA version via conda list | grep cuda.
================================================================================

CUDA SETUP: Something unexpected happened. Please compile from source:
git clone https://github.com/TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=117 make cuda11x
python setup.py install
CUDA SETUP: Setup Failed!
Traceback (most recent call last):
File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1099, in _get_module
return importlib.import_module("." + module_name, self.name)
File "/home/work/anaconda3/envs/kollava/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1006, in _find_and_load_unlocked
File "", line 688, in _load_unlocked
File "", line 883, in exec_module
File "", line 241, in _call_with_frames_removed
File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 32, in
from ...modeling_utils import PreTrainedModel
File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/transformers/modeling_utils.py", line 38, in
from .deepspeed import deepspeed_config, is_deepspeed_zero3_enabled
File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/transformers/deepspeed.py", line 37, in
from accelerate.utils.deepspeed import HfDeepSpeedConfig as DeepSpeedConfig
File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/accelerate/init.py", line 3, in
from .accelerator import Accelerator
File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/accelerate/accelerator.py", line 35, in
from .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_state
File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/accelerate/checkpointing.py", line 24, in
from .utils import (
File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/accelerate/utils/init.py", line 131, in
from .bnb import has_4bit_bnb_layers, load_and_quantize_model
File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/accelerate/utils/bnb.py", line 42, in
import bitsandbytes as bnb
File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/bitsandbytes/init.py", line 6, in
from . import cuda_setup, utils, research
File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/bitsandbytes/research/init.py", line 1, in
from . import nn
File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/bitsandbytes/research/nn/init.py", line 1, in
from .modules import LinearFP8Mixed, LinearFP8Global
File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/bitsandbytes/research/nn/modules.py", line 8, in
from bitsandbytes.optim import GlobalOptimManager
File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/bitsandbytes/optim/init.py", line 6, in
from bitsandbytes.cextension import COMPILED_WITH_CUDA
File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 20, in
raise RuntimeError('''
RuntimeError:
CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    _python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues_

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/work/anaconda3/envs/kollava/lib/python3.10/runpy.py", line 187, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/home/work/anaconda3/envs/kollava/lib/python3.10/runpy.py", line 110, in _get_module_details
import(pkg_name)
File "/home/work/testdataset1/KoLLaVA/llava/init.py", line 1, in
from .model import LlavaLlamaForCausalLM
File "/home/work/testdataset1/KoLLaVA/llava/model/init.py", line 1, in
from .language_model.llava_llama import LlavaLlamaForCausalLM, LlavaConfig
File "/home/work/testdataset1/KoLLaVA/llava/model/language_model/llava_llama.py", line 21, in
from transformers import AutoConfig, AutoModelForCausalLM,
File "", line 1075, in _handle_fromlist
File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1090, in getattr
value = getattr(module, name)
File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1089, in getattr
module = self._get_module(self._class_to_module[name])
File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1101, in _get_module
raise RuntimeError(
RuntimeError: Failed to import transformers.models.llama.modeling_llama because of the following error (look up to see its traceback):

    CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    _python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues_

전에도 오류가 계속 떴지만 이거랑은 다른 거였습니다. 그래서 아예 다시 git clone 하고, 가상환경도 새로 만들고 다 해봤는데, 현재는 이 오류만 계속 뜹니다. 다시 git clone하고, 가상환경도 다시 만드는데도 해결되지 않네요. 이런 에러는 주로 어떨 때 나오나요?

참고로 아나콘다는 제 리눅스 서버 환경에 Anaconda3-2020.11-Linux-x86_64.sh로 설치했습니다.

감사합니다.

Answer 1 · 2024-05-21T17:08:39.000Z

pip uninstall bitsandbytes 입력하고 나서 pip install bitsandbytes 하니까 해결됐습니다.

혹시 저같은 문제를 마주한 분들 있으면 도움이 되면 좋겠습니다. ㅎㅎ