chidiwilliams/buzz

I have cuda installed and new version of buzz, still it uses CPU rather than GPU

Walkerby opened this issue · 11 comments

Walkerby commented

My gpu is 4070ti. Cuda version is 12.5.1 on windows. Buzz version is 1.0.1.

raivisdejus commented

Whisper.cpp will use CPU, that is expected.

Can you test all of these:

  • Whisper
  • Faster Whisper
  • Huggingface for example with openai/whisper-small

Each of them have a different mechanism to enable CUDA, so curious if all of them do not work.

Also what CUDA version do you have?

Will try to figure this out...

Walkerby commented

Whisper.cpp will use CPU, that is expected.

Can you test all of these:

  • Whisper
  • Faster Whisper
  • Huggingface for example with openai/whisper-small

Each of them have a different mechanism to enable CUDA, so curious if all of them do not work.

Also what CUDA version do you have?

Will try to figure this out...

I tried whisper and faster whisper model. Both of them can only run on CPU. My cuda version is cuda_12.5.1_555.85_windows.

raivisdejus commented

@Walkerby Please see this #861
It has some notes on how you may be able to make it work while we figure out the official solution.

If you gt it working, please share some notes that may be helpful to others

raivisdejus commented

I was able to get CUDA GPU support working for Whisper and Huggingface whisper type. Tested on Windows server 2022 with CUDA 12.4

Install ffmpeg choco install ffmpeg

pip install ffmpeg
pip install ffmpeg-python

Install torch with CUDA support https://pytorch.org/get-started/locally/
f.e. pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124

Install appropriate buzz wheel from https://github.com/chidiwilliams/buzz/actions/runs/10221561439
f.e. pip install buzz_captions-1.0.2-cp312-cp312-win_amd64.whl

Run buzz python -m buzz and use some Huggingface model f.e. openai/whisper-large-v3

raivisdejus commented

See notes on GPU support on Windows here https://github.com/chidiwilliams/buzz/blob/main/CONTRIBUTING.md#gpu-support

And for best results use the latest development version from https://github.com/chidiwilliams/buzz/actions/workflows/ci.yml?query=branch%3Amain (to download log into the github)

Will close the issue, reopen if there are additional details