hallucinated words in the output

Question

hallucinated words in the output

Opened this issue 8 months ago · 10 comments

While running the program during silence periods irrelevant output like "okay" and "thank you" appears.
Is there a way to fix this or is it a feature of faster-whisper

Answer 1 · 2024-01-19T10:21:32.000Z

I have also noticed that behaviour.

I think the particular VAD model we're using, while conceptually it fits the project well, is almost useless at the moment, could be worth experimenting with other models.

For the time being, what I do is to read the language_probability. After a bit of experimenting, I've found out that setting a 0.9 threshold will basically prevent all the false positives.

websocketRecognition = new WebSocket(recognitionWebSocketAddress);
websocketRecognition.onmessage = function(event) {
    const response = JSON.parse(event.data);

    console.log(response.language_probability);

    if (response.language_probability > 0.9) {
        doSomethingWith(response.text);
    } else {
        console.log("Speech not recognized. Could be just noise or hallucinations");
    }
};

Answer 2 · 2024-02-02T08:41:42.000Z

@alesaccoia

But when I use this for only English, language_probability is always 1.

What is the solution in this case?

Answer 3 · 2024-02-02T08:43:37.000Z

I see, didn't try that. then maybe you could take a look at the word level probabilities to isolate the problem?

Answer 4 · 2024-02-02T08:51:49.000Z

I understnad, I looked at word level probabilities but couldn't find a threshold.

Answer 5 · 2024-02-02T08:52:36.000Z

So, do you suggest me to use multilanguage option?

Answer 6 · 2024-02-02T08:56:26.000Z

@alesaccoia
Also, when I use multilanguage mode, will I meet additional latency?

Answer 7 · 2024-02-02T16:33:18.000Z

Yes multiingual model does have additional latency, it first computes the audio language.
Although this is a very small extra latency compared to the transcription

Answer 8 · 2024-02-20T03:14:26.000Z

Please add "vad_filter=True" to fix the problem
Exam:
model.transcribe(fileName, beam_size=10, language="vi", vad_filter=True)

Answer 9 · 2024-02-20T06:59:51.000Z

@KZyred didn't know that argument existed. did you test that?

Answer 10 · 2024-02-20T11:45:00.000Z

Indeed it helps, however is not any miracle option.
It does decrease latency for long audios.
Other types of hallucination may appear, play with vad filter options.