Windows Large V3 Malloc 4GB Limitation

Question

Windows Large V3 Malloc 4GB Limitation

bijanbina opened this issue a month ago · 3 comments

Hi
On windows no matter how much mem you've got on ur machine you cannot get large v3 to run cuz malloc on windows has 4GB limit. would this be smt you can test and fix
Cheers

https://stackoverflow.com/questions/57881307/how-to-correctly-allocate-a-large-chunk-of-ram-with-calloc-malloc-in-windows-10

Answer 1 · 2024-05-06T22:54:45.000Z

Can you be more specific what is the issue that you're actually facing?
I tested Windows builds from MSYS2 environments, CLANG64 and UCRT64, and I do not see any immediate issues.

przemoc@NUC11PHKi7C002 CLANG64 /d/git/github.com/ggerganov/whisper.cpp
$ rm -rf build && cmake -B build && cmake --build build -j $(nproc)               
...

przemoc@NUC11PHKi7C002 CLANG64 /d/git/github.com/ggerganov/whisper.cpp
$ ./build/bin/main.exe -m models/ggml-large-v3.bin -f samples/jfk.wav
whisper_init_from_file_with_params_no_state: loading model from 'models/ggml-large-v3.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51866
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 1280
whisper_model_load: n_audio_head  = 20
whisper_model_load: n_audio_layer = 32
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 1280
whisper_model_load: n_text_head   = 20
whisper_model_load: n_text_layer  = 32
whisper_model_load: n_mels        = 128
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 5 (large v3)
whisper_model_load: adding 1609 extra tokens
whisper_model_load: n_langs       = 100
whisper_model_load:      CPU total size =  3094.36 MB
whisper_model_load: model size    = 3094.36 MB
whisper_init_state: kv self size  =  220.20 MB
whisper_init_state: kv cross size =  245.76 MB
whisper_init_state: compute buffer (conv)   =   36.26 MB
whisper_init_state: compute buffer (encode) =  926.66 MB
whisper_init_state: compute buffer (cross)  =    9.38 MB
whisper_init_state: compute buffer (decode) =  209.26 MB

system_info: n_threads = 4 / 8 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | METAL = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | CUDA = 0 | COREML = 0 | OPENVINO = 0

main: processing 'samples/jfk.wav' (176000 samples, 11.0 sec), 4 threads, 1 processors, 5 beams + best of 5, lang = en, task = transcribe, timestamps = 1 ...


[00:00:00.300 --> 00:00:09.000]   And so, my fellow Americans, ask not what your country can do for you, ask what you
[00:00:09.000 --> 00:00:11.000]   can do for your country.

whisper_print_timings:     load time =  1932.32 ms
whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:      mel time =    13.86 ms
whisper_print_timings:   sample time =    95.37 ms /   147 runs (    0.65 ms per run)
whisper_print_timings:   encode time = 32488.18 ms /     1 runs (32488.18 ms per run)
whisper_print_timings:   decode time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:   batchd time =  4972.74 ms /   145 runs (   34.29 ms per run)
whisper_print_timings:   prompt time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:    total time = 39508.21 ms

przemoc@NUC11PHKi7C002 UCRT64 /d/git/github.com/ggerganov/whisper.cpp
$ rm -rf build && cmake -B build && cmake --build build -j $(nproc)
...

przemoc@NUC11PHKi7C002 UCRT64 /d/git/github.com/ggerganov/whisper.cpp
$ ./build/bin/main.exe -m models/ggml-large-v3.bin -f samples/jfk.wav
whisper_init_from_file_with_params_no_state: loading model from 'models/ggml-large-v3.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51866
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 1280
whisper_model_load: n_audio_head  = 20
whisper_model_load: n_audio_layer = 32
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 1280
whisper_model_load: n_text_head   = 20
whisper_model_load: n_text_layer  = 32
whisper_model_load: n_mels        = 128
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 5 (large v3)
whisper_model_load: adding 1609 extra tokens
whisper_model_load: n_langs       = 100
whisper_model_load:      CPU total size =  3094.36 MB
whisper_model_load: model size    = 3094.36 MB
whisper_init_state: kv self size  =  220.20 MB
whisper_init_state: kv cross size =  245.76 MB
whisper_init_state: compute buffer (conv)   =   36.26 MB
whisper_init_state: compute buffer (encode) =  926.66 MB
whisper_init_state: compute buffer (cross)  =    9.38 MB
whisper_init_state: compute buffer (decode) =  209.26 MB

system_info: n_threads = 4 / 8 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | METAL = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | CUDA = 0 | COREML = 0 | OPENVINO = 0

main: processing 'samples/jfk.wav' (176000 samples, 11.0 sec), 4 threads, 1 processors, 5 beams + best of 5, lang = en, task = transcribe, timestamps = 1 ...


[00:00:00.300 --> 00:00:09.000]   And so, my fellow Americans, ask not what your country can do for you, ask what you
[00:00:09.000 --> 00:00:11.000]   can do for your country.

whisper_print_timings:     load time =  1010.02 ms
whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:      mel time =    21.38 ms
whisper_print_timings:   sample time =    85.51 ms /   147 runs (    0.58 ms per run)
whisper_print_timings:   encode time = 32120.25 ms /     1 runs (32120.25 ms per run)
whisper_print_timings:   decode time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:   batchd time =  4907.33 ms /   145 runs (   33.84 ms per run)
whisper_print_timings:   prompt time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:    total time = 38150.42 ms