chinese characters not showing up on windows
andrewchen5678 opened this issue · 2 comments
andrewchen5678 commented
I tried the 1.5.4 build (don't know how to compile the latest) on windows to transcribe Chinese language audios, but it is getting garbage instead of unicode:
(transcribe-audio-py3.12) PS C:\Users\andrew\codes\transcribe_audio> .\whispercpp-bin-x64\main.exe -m .\whisper_models\ggml-large-v3.bin -f R:\andrewdata\audio_test\happydailytest2_16k.wav -l yue
whisper_init_from_file_with_params_no_state: loading model from '.\whisper_models\ggml-large-v3.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab = 51866
whisper_model_load: n_audio_ctx = 1500
whisper_model_load: n_audio_state = 1280
whisper_model_load: n_audio_head = 20
whisper_model_load: n_audio_layer = 32
whisper_model_load: n_text_ctx = 448
whisper_model_load: n_text_state = 1280
whisper_model_load: n_text_head = 20
whisper_model_load: n_text_layer = 32
whisper_model_load: n_mels = 128
whisper_model_load: ftype = 1
whisper_model_load: qntvr = 0
whisper_model_load: type = 5 (large v3)
whisper_model_load: adding 1609 extra tokens
whisper_model_load: n_langs = 100
whisper_model_load: CPU buffer size = 3094.86 MB
whisper_model_load: model size = 3094.36 MB
whisper_init_state: kv self size = 220.20 MB
whisper_init_state: kv cross size = 245.76 MB
whisper_init_state: compute buffer (conv) = 32.42 MB
whisper_init_state: compute buffer (encode) = 212.42 MB
whisper_init_state: compute buffer (cross) = 9.38 MB
whisper_init_state: compute buffer (decode) = 99.24 MB
system_info: n_threads = 4 / 6 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | METAL = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 0 | VSX = 0 | CUDA = 0 | COREML = 0 | OPENVINO = 0 |
main: processing 'R:\andrewdata\audio_test\happydailytest2_16k.wav' (43665060 samples, 2729.1 sec), 4 threads, 1 processors, 5 beams + best of 5, lang = yue, task = transcribe, timestamps = 1 ...
[00:00:00.000 --> 00:00:09.560] 過呢個所謂嘅概念股曾經都跌好多呢係嗰個原因囉係咁大家就要即係小心啲去區別
[00:00:09.560 --> 00:00:13.200] 即係導資嘅跌同蠟燭跌係唔通嘅就係咁簡單
[00:00:13.200 --> 00:00:18.000] 好我哋頭先冇咩問咩股票呀唔問你遲有啲咩
[00:00:18.000 --> 00:00:21.000] 今日唔該細魚浩然同我哋分析大事嘅下星期二周我問你拜拜
[00:00:21.000 --> 00:00:22.000] 好唔介你拜拜
(transcribe-audio-py3.12) PS C:\Users\andrew\codes\transcribe_audio>
but on mac it shows up fine
(transcribe-audio-py3.12) it3@it3s-MacBook-Pro-2 transcribe_audio % ./bin/whisper-darwin-arm64 -m ./whisper_models/ggml-large-v3.bin -f /Volumes/andrewdata/audio_test/happydailytest2_16k.wav -l yue
whisper_init_from_file_with_params_no_state: loading model from './whisper_models/ggml-large-v3.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab = 51866
whisper_model_load: n_audio_ctx = 1500
whisper_model_load: n_audio_state = 1280
whisper_model_load: n_audio_head = 20
whisper_model_load: n_audio_layer = 32
whisper_model_load: n_text_ctx = 448
whisper_model_load: n_text_state = 1280
whisper_model_load: n_text_head = 20
whisper_model_load: n_text_layer = 32
whisper_model_load: n_mels = 128
whisper_model_load: ftype = 1
whisper_model_load: qntvr = 0
whisper_model_load: type = 5 (large v3)
whisper_model_load: adding 1609 extra tokens
whisper_model_load: n_langs = 100
whisper_backend_init: using Metal backend
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M2 Max
ggml_metal_init: picking default device: Apple M2 Max
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil
ggml_metal_init: error: could not use bundle path to find ggml-metal.metal, falling back to trying cwd
ggml_metal_init: loading 'ggml-metal.metal'
ggml_metal_init: error: Error Domain=NSCocoaErrorDomain Code=260 "The file “ggml-metal.metal” couldn’t be opened because there is no such file." UserInfo={NSFilePath=ggml-metal.metal, NSUnderlyingError=0x600001754780 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}}
whisper_backend_init: ggml_backend_metal_init() failed
whisper_model_load: CPU total size = 3094.36 MB
whisper_model_load: model size = 3094.36 MB
whisper_backend_init: using Metal backend
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M2 Max
ggml_metal_init: picking default device: Apple M2 Max
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil
ggml_metal_init: error: could not use bundle path to find ggml-metal.metal, falling back to trying cwd
ggml_metal_init: loading 'ggml-metal.metal'
ggml_metal_init: error: Error Domain=NSCocoaErrorDomain Code=260 "The file “ggml-metal.metal” couldn’t be opened because there is no such file." UserInfo={NSFilePath=ggml-metal.metal, NSUnderlyingError=0x6000017579c0 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}}
whisper_backend_init: ggml_backend_metal_init() failed
whisper_init_state: kv self size = 220.20 MB
whisper_init_state: kv cross size = 245.76 MB
whisper_init_state: compute buffer (conv) = 36.26 MB
whisper_init_state: compute buffer (encode) = 934.34 MB
whisper_init_state: compute buffer (cross) = 9.38 MB
whisper_init_state: compute buffer (decode) = 209.26 MB
system_info: n_threads = 4 / 12 | AVX = 0 | AVX2 = 0 | AVX512 = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | METAL = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | CUDA = 0 | COREML = 0 | OPENVINO = 0
main: processing '/Volumes/andrewdata/audio_test/happydailytest2_16k.wav' (43665060 samples, 2729.1 sec), 4 threads, 1 processors, 5 beams + best of 5, lang = yue, task = transcribe, timestamps = 1 ...
[00:00:00.000 --> 00:00:09.560] 過呢個所謂嘅概念股曾經都跌好多呢係嗰個原因囉係咁大家就要即係小心啲去區別
[00:00:09.560 --> 00:00:13.200] 即係導資嘅跌同蠟燭跌係唔通嘅就係咁簡單
[00:00:13.200 --> 00:00:18.000] 好我哋頭先冇咩問咩股票呀唔問你遲有啲咩
[00:00:18.000 --> 00:00:21.000] 今日唔該細魚浩然同我哋分析大事嘅下星期二周我問你拜拜
[00:00:21.000 --> 00:00:22.000] 好唔介你拜拜
bobqianic commented
chcp 65001
jason-ni commented
chcp 65001
Just encountered the same issue today. @andrewchen5678 https://stackoverflow.com/questions/57131654/using-utf-8-encoding-chcp-65001-in-command-prompt-windows-powershell-window
It's not an issue of whisper.cpp whatever.