For call in sample "microsoft/phi-2" "python gradio_chat.py --baseonly" need nvidia-cuda-toolkit

[Serg2DFX]

use project template:

I do as in the example:

conda activate phi-2-env python gradio_chat.py --baseonly

and get the result

it helped me to install:
sudo apt install nvidia-cuda-toolkit

It is worth adding to the documentation (README.MD) or need check in prepare configuration ?

[elsaco]

@Serg2DFX the plugin will install CUDA for you and all the requirements for the phi-2 model. If there were missing libraries when you run python -m bitsandbytes then check your installation. While you could install CUDA outside of the windows-ai-studio I find it safer to just let the plugin do it.

This is output after activating the phi-2-env and loading bitsandbytes module:

(phi-2-env) root@texas:/mnt/e/ai/phi-2/inference# python -m bitsandbytes
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++ BUG REPORT INFORMATION ++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

+++++++++++++++++++ ANACONDA CUDA PATHS ++++++++++++++++++++
/opt/miniconda/envs/phi-2-env/nsight-compute/2023.3.1/target/linux-desktop-t210-a64/libcuda-injection.so
/opt/miniconda/envs/phi-2-env/nsight-compute/2023.3.1/target/linux-desktop-glibc_2_19_0-ppc64le/libcuda-injection.so
/opt/miniconda/envs/phi-2-env/nsight-compute/2023.3.1/target/linux-desktop-glibc_2_11_3-x64/libcuda-injection.so
/opt/miniconda/envs/phi-2-env/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda120.so
/opt/miniconda/envs/phi-2-env/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda115_nocublaslt.so
/opt/miniconda/envs/phi-2-env/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda114_nocublaslt.so
/opt/miniconda/envs/phi-2-env/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda117_nocublaslt.so
/opt/miniconda/envs/phi-2-env/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda118_nocublaslt.so
/opt/miniconda/envs/phi-2-env/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda114.so
/opt/miniconda/envs/phi-2-env/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda117.so
/opt/miniconda/envs/phi-2-env/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda118.so
/opt/miniconda/envs/phi-2-env/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda121.so
/opt/miniconda/envs/phi-2-env/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda121_nocublaslt.so
/opt/miniconda/envs/phi-2-env/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda111.so
/opt/miniconda/envs/phi-2-env/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda110.so
/opt/miniconda/envs/phi-2-env/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda120_nocublaslt.so
/opt/miniconda/envs/phi-2-env/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda122.so
/opt/miniconda/envs/phi-2-env/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda115.so
/opt/miniconda/envs/phi-2-env/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda110_nocublaslt.so
/opt/miniconda/envs/phi-2-env/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda111_nocublaslt.so
/opt/miniconda/envs/phi-2-env/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda122_nocublaslt.so
/opt/miniconda/envs/phi-2-env/lib/python3.9/site-packages/onnxruntime/capi/libonnxruntime_providers_cuda.so
/opt/miniconda/envs/phi-2-env/lib/python3.9/site-packages/torch/lib/libc10_cuda.so
/opt/miniconda/envs/phi-2-env/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so
/opt/miniconda/envs/phi-2-env/lib/python3.9/site-packages/torch/lib/libtorch_cuda_linalg.so
/opt/miniconda/envs/phi-2-env/lib/libcudart.so
/opt/miniconda/envs/phi-2-env/lib/stubs/libcuda.so

++++++++++++++++++ /usr/local CUDA PATHS +++++++++++++++++++


+++++++++++++++ WORKING DIRECTORY CUDA PATHS +++++++++++++++


++++++++++++++++++ LD_LIBRARY CUDA PATHS +++++++++++++++++++

++++++++++++++++++++++++++ OTHER +++++++++++++++++++++++++++
COMPILED_WITH_CUDA = True
COMPUTE_CAPABILITIES_PER_GPU = ['8.6']
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++ DEBUG INFO END ++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Running a quick check that:
    + library is importable
    + CUDA function is callable


WARNING: Please be sure to sanitize sensible info from any such env vars!

SUCCESS!
Installation was successful!

Gradio Chat output:

(phi-2-env) root@texas:/mnt/e/ai/phi-2/inference# python gradio_chat.py --baseonly
Number of GPUs available: 1
Running on device: cuda
CPU threads: 3
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████| 2/2 [01:57<00:00, 58.59s/it]
Number of GPUs available: 1
Model ../model-cache/microsoft/phi-2 loaded successfully on cuda
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.

[Serg2DFX]

it helped me to install:
sudo apt install nvidia-cuda-toolkit