modelscope/FunCodec

Funcodec进行解码处理时，波形幅值不一致

huhuqwaszxedc opened this issue · 2 comments

huhuqwaszxedc commented

感谢您的分享，我在尝试进行Funcodec解码时，忽略了_scale_输出后，波形结构相近，但是幅值差异较大，请问我该如何修改代码。

speech2token = Speech2Token("egs/LibriTTS/codec/exp/audio_codec-encodec-zh_en-general-16k-nq32ds320-pytorch/config.yaml", "egs/LibriTTS/codec/exp/audio_codec-encodec-zh_en-general-16k-nq32ds320-pytorch/model.pth", sampling_rate=16000)

audio, rate = librosa.load("egs/LibriTTS/codec/test_wav/BAC009S0002W0122.wav", sr=16000) 
audio_32 = np.reshape(audio, (1,1,-1))
output = speech2token(audio_32, bit_width=16000, run_mod="encode")
tokens = output[0][0]
tokens_t = tokens.permute(1, 2, 0)
audio_re = speech2token(tokens_t, bit_width=16000, run_mod="decode")
image
ZhihaoDU commented

Currently, FunCodec doesn't keep the amplitude information, you can restore it by yourself as follows:

speech2token = Speech2Token("egs/LibriTTS/codec/exp/audio_codec-encodec-zh_en-general-16k-nq32ds320-pytorch/config.yaml", "egs/LibriTTS/codec/exp/audio_codec-encodec-zh_en-general-16k-nq32ds320-pytorch/model.pth", sampling_rate=16000)

audio, rate = librosa.load("egs/LibriTTS/codec/test_wav/BAC009S0002W0122.wav", sr=16000) 
# save the volume of input audio
volume = np.sqrt(np.mean(np.square(audio)))
audio_32 = np.reshape(audio, (1,1,-1))
output = speech2token(audio_32, bit_width=16000, run_mod="encode")
tokens = output[0][0]
tokens_t = tokens.permute(1, 2, 0)
audio_re = speech2token(tokens_t, bit_width=16000, run_mod="decode")
# restore the volume
audio_re = audio_re * volume
huhuqwaszxedc commented

感谢您的指导！