4_0, 4_1, and 8_0 Quantized models fail to load - SIGSEGV invalid memory reference
sigaloid opened this issue · 3 comments
I've quantized all of my models with the quantize-all script and this led me to find out a large portion of quantized models don't load properly in whisper-rs. Any model quantized at q4_0, q4_1, or q8_0 do not load and throw a SIGSEGV invalid memory reference. Oddly, ggml-large-v1-q4_0.bin throws a proper error but the rest of them (that I tested) core-dumped.
These work fine in whisper.cpp when running in ./main.
Attached is a ggml-tiny-q4_0.bin that fails to load with the error. Quantized with
./quantize models/ggml-tiny.bin models/ggml-tiny-q4_0.bin q4_0I actually hit this exact issue, and found the source of it: ggerganov/whisper.cpp#993
A workaround for now is cloning this repo, then updating the git submodule of whisper.cpp to point to the latest upstream instead of the current release tag it's fixed at.
Ok, great to hear that it's patched upstream. thanks!