Windows process crashes when the GPU model is unloaded

Question

Windows process crashes when the GPU model is unloaded

satisl opened this issue 2 years ago · 48 comments

Thanks for your work first. It's useful.
Howerver, there's still something wrong. It returns -1073740791 (0xC0000409) when dealing with a audio file in chinese. I have defined a function in which the variable 'result' is used to accept the 'segments' returned by the fast-whisper. It's normal in this function, but abnormal after being returned by the function.

The line 'print(result)' works.

But after the result is returned, python returns -1073740791 (0xC0000409) and terminates
When changing the model or the language, it went properly.
Confused.

Answer 1 · 2023-03-23T14:15:26.000Z

The same error is reported in #64 and is related to the cuDNN installation. Can you check that?

Answer 2 · 2023-03-23T14:26:08.000Z

Really? But I have already installed Cudnn according to the Nivida documentation, and if I use medium-ct2 instead of large-v2-ct2, switch languages, or process other files, this type of problem will not occur.

Answer 3 · 2023-03-23T14:29:29.000Z

How much VRAM does your GPU have?

Answer 4 · 2023-03-23T14:30:41.000Z

about 6000Mb and 4000-5000Mb when running

Answer 5 · 2023-03-23T14:37:33.000Z

Possibly you are running out of memory for this specific file. Can you try using compute_type="int8_float16"?

Answer 6 · 2023-03-23T14:43:02.000Z

It still didn't work.
When I tried using compute_type = "float32", it failed and returned "cuda out of memory".
Howerver, this time it returned nothing but -1073740791 (0xC0000409).
Seems like that it's not the reason.

I convert the audio file's format from aac to mp3. But it still didn't work.
Seems like that it's not the format that's wrong.

Answer 7 · 2023-03-23T19:47:16.000Z

Is it possible for you to share this audio file?

Answer 8 · 2023-03-24T01:45:04.000Z

the video file
with command
"ffmpeg -i "video file" -vn -ar 16000 "aac file""
"ffmpeg -i "video file" -vn -c:a libmp3lame -ar 16000 "mp3 file""
i extract the audio from the video

Answer 9 · 2023-03-25T18:17:10.000Z

I don't reproduce the error on Windows 11 with CUDA 11.8.0 and cuDNN 8.8.1.

Are you using the latest version of faster-whisper?

Answer 10 · 2023-03-26T01:11:10.000Z

The first thing I do when this error occurred is to update the faster-whisper

Answer 11 · 2023-03-26T02:12:13.000Z

Seems like that it's difficult to reproduce the error. Since this problem only occurs with this unique file under certain conditions (I have also processed various other files since then, and the result is normal operation). Perhaps this issue can be put on hold?
I will reopen this issue if the same error occurs in other files. Thank you for your help in so many days.

Answer 12 · 2023-03-27T03:05:38.000Z

By coincidence, I discovered the cause of the error. When the model is unloaded, the program will crash.
Previously, model was a local variable and would automatically unload the model when the function ended. And the program crashes.
Now, model is a global variable and would automatically unload the model when the program ended.

Though the program will crash when unloading the model, it will happen after all things are finished now.

Howerver, I have no idea why the program will crash when unloading the model.

Answer 13 · 2023-03-27T12:11:25.000Z

Does it also crash when you manually unload the model with del model?

Answer 14 · 2023-03-27T12:24:44.000Z

No, it doesn't.

It will when processing certain files (five out of approximately ninety files). If this error occurs when it processes a file, it will occur no matter how many times it processes the file.
My environment is rtx 3060 laptop, windows 10, cuda11.7, cudnn8.8.0, python3.10.10, model large-v2, language 'zh'.
Attempts have been made to reinstall the python environment or change the python version to 3.9.16, but this type of issue still exists.
The model have been redownloaded, but issue exits.

Answer 15 · 2023-03-27T12:35:42.000Z

If the model is manually unloaded after processing the file, the program will crash

Answer 16 · 2023-04-10T08:06:28.000Z

I found a way to run the transcription in a separate process so that even though it exits that child process, it doesn't exit your main script. Here is a working example:

from multiprocessing import Process, Manager
from faster_whisper import WhisperModel

def work_log(argsList, returnList):
    model_size = "large-v2"
    model = WhisperModel(model_size, device="cuda", compute_type="float16")
    segments, info = model.transcribe(*argsList, beam_size=5)
    returnList[0] = [list(segments), info]

# workaround to deal with a termination issue: https://github.com/guillaumekln/faster-whisper/issues/71
def runWhisperSeperateProc(*args):
    with Manager() as manager:
        returnList = manager.list([None])
        p = Process(target=work_log, args=[args, returnList])  # add return target to end of args list
        p.start()
        p.join()
        p.close()
        return returnList[0]

if __name__ == '__main__':
    segments, info = runWhisperSeperateProc("audio.mp3")
    print(segments, info)

Answer 17 · 2023-04-23T16:16:50.000Z

same issues

Answer 18 · 2023-04-27T18:30:58.000Z

I found a way to run the transcription in a separate process so that even though it exits that child process, it doesn't exit your main script. Here is a working example:

from multiprocessing import Process, Manager
from faster_whisper import WhisperModel

def work_log(argsList, returnList):
    model_size = "large-v2"
    model = WhisperModel(model_size, device="cuda", compute_type="float16")
    segments, info = model.transcribe(*argsList, beam_size=5)
    returnList[0] = [list(segments), info]

# workaround to deal with a termination issue: https://github.com/guillaumekln/faster-whisper/issues/71
def runWhisperSeperateProc(*args):
    with Manager() as manager:
        returnList = manager.list([None])
        p = Process(target=work_log, args=[args, returnList])  # add return target to end of args list
        p.start()
        p.join()
        p.close()
        return returnList[0]

if __name__ == '__main__':
    segments, info = runWhisperSeperateProc("audio.mp3")
    print(segments, info)

same issue and open a Process to run works for me

Answer 19 · 2023-04-27T18:39:35.000Z

@ProjectEGU @yslion @DoodleBears Are you all using the library on Windows?

Answer 20 · 2023-04-27T19:36:13.000Z

I can now reproduce the issue on Windows.

It is somehow related to the temperature fallback. Can you try setting temperature=0?

Answer 21 · 2023-04-28T04:25:36.000Z

Glad to know that the reason has been detected. With this setting, the program runs nomally.
However, maybe it will produce slightly worsen result? I don't know.

Answer 22 · 2023-04-28T04:41:02.000Z

@ProjectEGU @yslion @DoodleBears Are you all using the library on Windows?

Yes, I am using the library on Windows 10, I will try temperature=0 this evening

Answer 23 · 2023-04-28T12:49:23.000Z

I have a possible fix for this issue in OpenNMT/CTranslate2#1201, but I can't test on my Windows machine today. Can you help testing?

Go to the build page
Download the artifact "python-wheels"
Extract the archive
Install the Windows wheel matching your Python version with pip install --force-reinstall <wheel file>

Answer 24 · 2023-04-28T14:19:03.000Z

I have a possible fix for this issue in OpenNMT/CTranslate2#1201, but I can't test on my Windows machine today. Can you help testing?

Go to the build page

Download the artifact "python-wheels"

Extract the archive

Install the Windows wheel matching your Python version with pip install --force-reinstall <wheel file>

Yes, I will try it now, by the way I try temperature=0, it works (process did not exit)

Answer 25 · 2023-04-28T15:17:38.000Z

I have a possible fix for this issue in OpenNMT/CTranslate2#1201, but I can't test on my Windows machine today. Can you help testing?

Go to the build page

Download the artifact "python-wheels"

Extract the archive

Install the Windows wheel matching your Python version with pip install --force-reinstall <wheel file>

I install the wheel.

try to run without temperature=0 —— same issue (process still exit)
try to run with temperature=0 —— works

Answer 26 · 2023-04-28T15:36:14.000Z

when using temperature=0:
I met segmentation fault BEFORE installing the wheel sometimes when I call the function below many times, don't know why

Sorry for did not keep the log, I remember it mentioned: cuBLAS and CUDA ...... segments fault, once I reproduce it, I will share the error log

def transcribe_speeches(self):
    log.init_logging(debug=True)
    # NOTE: 读取音频文件
    logger.info(f"开始语音转文字")
    whisper = WhisperModel(WHISPER_MODEL, device="cuda", compute_type="float16")
    speeches_num = len(self.speeches)
    for index, speech in enumerate(self.speeches):
        logger.debug(f"开始识别 {speech.audio_path}")
        speech_text = ''
        # NOTE: 识别音频文件
        segments, _ = whisper.transcribe(
            audio=speech.audio_path,
            language='zh',
            vad_filter=False,
            temperature=0,
            initial_prompt='以下是普通话的句子。'
            )
        segments = list(segments)
        if len(segments) == 0:
            logger.warning(f"识别结果为空: {speech.audio_path}")
        else:
            speech_text = '，'.join([segment.text for segment in segments])
            logger.info(f"识别结果({index+1}/{speeches_num}): {speech_text}")
        self.speeches[index].text = speech_text
        
    
    logger.info(f"结束语音转文字: {self.speeches}")
    # queue.put(self.speeches)
    # FIXME: 卸载模型后会导致程序终止
    del whisper

Answer 27 · 2023-04-28T15:43:46.000Z

import os
import subprocess
from faster_whisper import WhisperModel

files = os.listdir('input')
for file in files:
    file = file.rsplit('.', 1)[0]
    print(file)
    command = f'ffmpeg -i \"input\\{file}.mp4\" -vn -ar 16000 -c:a libmp3lame -y \"tmp\\{file}.mp3\"'
    subprocess.run(command, capture_output=True, check=True)
    model = WhisperModel('large-v2', device='cuda', compute_type='float16')
    segment, info = model.transcribe(f'tmp\\{file}.mp3', language=None)
    for i in segment:
        text = i.text
    del model

print('finished')

same error

Answer 28 · 2023-04-28T15:48:46.000Z

Thank you for the test! I will keep looking...

Answer 29 · 2023-05-06T12:54:54.000Z

I'm not sure if this is directly related, but I get Segmentation fault error from time to time when I start to analyze the transcription segments via for segment in segments: .... I can't really pin down the precise location but it must be somewhere in WhisperModel.generate_segments and it happens only when my program tries to handle some remaining chunks at the end of a stream that are basically background noise.
Since I set temperature=0 it hasn't happened again.

Answer 30 · 2023-05-10T06:53:30.000Z

@fquirin Are you also running on Windows with a GPU? If not, I’m not sure your issue is related. You can open another issue if you can share the audio and options triggering the crash.

Answer 31 · 2023-05-10T11:29:10.000Z

@guillaumekln I'm running it on Windows + CPU.
The problem is I can't reproduce it with audio files so far, only with my live-streaming server, but a pretty reliable way to get the segmentation fault is coughing 🤔. First I thought it was a problem with my code but it never happens with temperature=0 and so far it never happend on Linux Aarch64 as well (with or without temp=0). Notably another difference to my Linux Aarch64 system is that my x86 CPU is much much faster (maybe a race condition?).

Btw, when I run my "coughing" test WAV files I noticed that Whisper can start to hallucinate pretty extensively with temperature != 0.

I'll try to pin down the segmentation fault by adding some debug info to WhisperModel.generate_segments

Answer 32 · 2023-05-16T05:50:04.000Z

I have the same error when running the script in windows 10 WSL (ubuntu)

edit: i installed all the deps and tried again and it worked now

Answer 33 · 2023-06-01T00:42:45.000Z

same issue with windows 11

Answer 34 · 2023-06-07T12:25:06.000Z

I was able to avoid that error with the temperature=0 setting. Will this setting adversely affect the transcribe results? I searched the whisper repo, but couldn't find a satisfactory answer.

Answer 35 · 2023-07-07T15:55:00.000Z

Yes disabling the temperature fallback can affect the results. The fallback is mostly useful to recover from cases where the model generates the same token in a loop.

Answer 36 · 2023-07-08T01:10:32.000Z

Yes disabling the temperature fallback can affect the results. The fallback is mostly useful to recover from cases where the model generates the same token in a loop.

Thank you. My test results were the same as you said.

Answer 37 · 2023-07-14T17:12:51.000Z

My runtime environment is Python 3.11.4, CUDA 11.8.0, graphics card driver 522.06, and cudnn-windows-x86_64-8.9.3.28. I am using the faster-whisper project, and when I try to load the model using GPU, Python returns -1073740791 (0xC0000409) error. However, when I use CPU, the error does not occur.

I have tried various solutions, including the ones you mentioned above, such as installing CUDA environment, adding system variables, and modifying the temperature to 0. None of them have worked.

Whenever I iterate over the segments, CUDA crashes, and the program terminates.

Finally, when I test and print print(torch.cuda.is_available()) to check if CUDA device is recognized as True, the program runs without any issues.

My personal estimation is that there might be an issue with the initialization and release of CUDA in CT2.

Answer 38 · 2023-07-14T19:25:10.000Z

My runtime environment is Python 3.11.4, CUDA 11.8.0, graphics card driver 522.06, and cudnn-windows-x86_64-8.9.3.28. I am using the faster-whisper project, and when I try to load the model using GPU, Python returns -1073740791 (0xC0000409) error. However, when I use CPU, the error does not occur.

I have tried various solutions, including the ones you mentioned above, such as installing CUDA environment, adding system variables, and modifying the temperature to 0. None of them have worked.

Whenever I iterate over the segments, CUDA crashes, and the program terminates.

Finally, when I test and print print(torch.cuda.is_available()) to check if CUDA device is recognized as True, the program runs without any issues.

My personal estimation is that there might be an issue with the initialization and release of CUDA in CT2.

Go check if you install zlib refer to https://docs.nvidia.com/deeplearning/cudnn/install-guide/index.html#install-zlib-windows

Answer 39 · 2023-07-15T10:31:59.000Z

@JamePeng This is a different issue. The issue described in this thread is a crash when unloading the model.

The error you get generally means that the program cannot locate the cuDNN and/or Zlib libraries. There are already several discussions about this.

Answer 40 · 2023-07-16T08:56:23.000Z

@guillaumekln ok, it worked now, thanks for your help

Answer 41 · 2023-08-11T07:59:55.000Z

Do we have any updates on resolving this issue? Currently, using the workaround of setting temperature=0 is an option, but it could potentially impact the model's performance.

Answer 42 · 2023-08-14T07:06:24.000Z

Does it also crash when you manually unload the model with del model?

I compileted my app with nuitka, and then run it as Administrastor User , it will not crash when unload model.

Answer 43 · 2023-11-07T19:51:29.000Z

I have the same problem. My config python 3.10.7 CUDA ToolKit 11.8 cuDNN 8.9.6 and add to PATH. If i change temperature=0, i get looping

Answer 44 · 2023-12-10T18:51:59.000Z

i had the same problem and i think i fixed it in my case by moving the faster whisper import inside the function that needs/uses it.

But, keep in mind that I am using faster whisper through stable whisper, and i need to import some stuff from the faster whisper library. I previously imported it globally in the top and found that my app will sometimes crashes after loading and reloading different model, but then after moving it to only inside the function that uses it somehow the crash is gone

Answer 45 · 2024-05-09T14:33:55.000Z

I have the same issue. I reinstall pytorch with this command conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia and solve the problem.

Answer 46 · 2024-07-18T10:32:01.000Z

I can consistently reproduce it with the latest master, Python 3.11.1 and Cuda 12.5 on Windows 10, 3 minutes of audio and tiny model, with the following simple code:

from faster_whisper import WhisperModel
model = WhisperModel("tiny", device="cuda", compute_type="auto")
segments, info = model.transcribe("js.wav")
for segment in segments:
    print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))

I only installed Cuda Toolkit 12.5, not cuDNN. At no point does the system max out on CPU, GPU or memory.

If device="cpu" is forced, the issue does not occur, nor does it with temperature=0.0 as stated above. Curiously, it also does not occur if I don't iterate to the end of segments generator: in my case, if the iteration is stopped roughly half way through, there is no crash.

If I put del model at the end, on some occasions the crash comes on that instruction, but sometimes after it.