Integrating Tacotron and LPCNet: Training tacotron with .f32 features

Question

Integrating Tacotron and LPCNet: Training tacotron with .f32 features

rpratesh opened this issue 5 years ago · 31 comments

In the ReadMe, it's mentioned

Convert the data generated at the last step which has .f32 extension to what could be loaded with numpy. I merge it to the Tacotron feeder here and here with the following code.

> mel_target = np.fromfile(os.path.join(self._mel_dir, meta[0]), dtype='float32')
> mel_target = np.resize(mel_target, (-1, self._hparams.num_mels))

But, meta[0] will have speech-audio-xxxx.npy files while self._mel_dir would have speech-mel-xxxx.npy files. So, the above code snippet is trying to search for speech (npy or f32) files inside mel_dir. Is there any thing wrong in the above code snippet.

One more doubt: Where should I copy the .f32 file generated in previous step, in Mels or in wavs or in Linear folder so that we can train Tacotron with these features generated.

Also, In this case, should I use

python train.py --model='Tacotron-2'

which trains entire tacotron+wavenet

or use

python train.py --model='Tacotron'

which trains only Tacotron.

Thanks

Answer 1 · 2019-05-17T06:36:45.000Z

about meta[0].
If you use tacotron2 preprocessing, you will get three folders(audio, melsprectrum and linear spectrum) and a txt file used to provide the info. Because I do not need audio file in tacotron2 training, so I make a soft link of audio folder to the f32 folder. Meanwhile, I can training tacotron2 with mels conventionally. Of course, I should modify the first column(meta[0]) to the real name of f32 file. You can modify it to your actual path of f32 according to your situation.
only train tacotron. you have another vocoder LPCNet instead of wavenet.

Answer 2 · 2019-05-17T09:32:25.000Z

So while training tacatron2 , i should replace/softlink the f32 to audio folder of training_data( after preprocessing) and train.txt first column(meta[0]) should be actual name of f32 files right?
if above is taken care below lines should be added to feeder.py.

mel_target = np.fromfile(os.path.join(self._audio_dir, meta[0]), dtype='float32')
mel_target = np.resize(mel_target, (-1, self._hparams.num_mels))

Basically what npy should be loaded in feeder audio or melspectrum .?

Answer 3 · 2019-05-17T10:00:50.000Z

of course, you can do that as long as your path is the f32.file .

Answer 4 · 2019-05-24T08:07:13.000Z

Hi, the feature extracted with feature_extract.sh script is saved as .f32 file and then it's used to train Tacotron2 . But Normally Tacotron2 was used to predict mel spectrogram. Here, T2 + LPCNet ,is the predict target of T2 is changed or just replace mel spectrogram with .f32 feature?

Answer 5 · 2019-05-24T08:10:19.000Z

@superhg2012 replace the f32 created to audio folder..
check this diff based on Mlwoo changes.
https://github.com/alokprasad/LPCTron/blob/master/Tacotron-2/Tacotron2-lpcnet_changes.diff

Answer 6 · 2019-05-24T08:27:52.000Z

@alokprasad thanks a lot !! #

Answer 7 · 2019-06-13T09:24:51.000Z

@alokprasad can you post your samples?

Answer 8 · 2019-06-13T10:50:21.000Z

@superhg2012
Please find the recording,,( They are not good)
I think we should retrain lpcnet with f32 generated from tactron2
https://vocaroo.com/i/s1Dx9nbKFeuY
https://vocaroo.com/i/s1VRBWayVzrD

Answer 9 · 2019-06-13T11:21:57.000Z

@alokprasad I can not reach the link you posted, please refer to [#1], (#1) posted audio sample, it seems that the author did not use GTA training mode.

Answer 10 · 2019-06-13T14:42:03.000Z

@alokprasad You have a lot work to do because you should calculate the length of audio according to the number of frames and add the tail to the audio. We did not use GTA mode because the job is trivial.
LPCNet is sensitive to pitch params. I think gta mode will result in one deviation of baseline to another if t2 is not trained well.

Answer 11 · 2019-06-14T13:23:27.000Z

@MlWoo do you mean that each Audio file should be of same length or it should in integral multiple of frames?

Answer 12 · 2019-06-15T01:23:55.000Z

@alokprasad more work. LPCnet will cut off the silence of audio in default, you should modify LPCnet code to cooperate with gta result of T2.

Answer 13 · 2019-07-11T07:22:54.000Z

@MlWoo Can you point to code in LPCNet , where the modification need to be done.

Answer 14 · 2019-07-11T08:04:01.000Z

@MlWoo i saw that xiph@554b6df there is silence removal here, and this is needed only to
during training of LPCnet .
Should i remove this code and Train LPCNet.

Answer 15 · 2019-07-15T05:28:26.000Z

@MlWoo Can we add this in Tacotron training for silence removal
gooofy/zamia-tts@66bd10d

Answer 16 · 2019-08-08T03:13:09.000Z

@alokprasad Tacotron training with silence removal is maybe a good idea when training English. It is bad idea while training Chinese(mandarin) because the very short silence is benefiticial to the prosody. I am not very sure about that is good to English cauz' I am not a native English speaker. Removing the long silence at the beginning and end of an audio is necessary when training Tacotron2.

Answer 17 · 2019-08-20T12:08:59.000Z

@superhg2012 replace the f32 created to audio folder..
check this diff based on Mlwoo changes.
https://github.com/alokprasad/LPCTron/blob/master/Tacotron-2/Tacotron2-lpcnet_changes.diff

I see you save audio (by preprocess) to meta[0], so you use audio as mel_target to train Tacotron2?

Answer 18 · 2019-08-20T12:10:12.000Z

Hi, the feature extracted with feature_extract.sh script is saved as .f32 file and then it's used to train Tacotron2 . But Normally Tacotron2 was used to predict mel spectrogram. Here, T2 + LPCNet ,is the predict target of T2 is changed or just replace mel spectrogram with .f32 feature?

Is the way to train T2? I don't understand well.

Answer 19 · 2019-08-26T08:15:57.000Z

Hi, the feature extracted with feature_extract.sh script is saved as .f32 file and then it's used to train Tacotron2 . But Normally Tacotron2 was used to predict mel spectrogram. Here, T2 + LPCNet ,is the predict target of T2 is changed or just replace mel spectrogram with .f32 feature?

Is the way to train T2? I don't understand well.

I see, we use f32 to train T2 instead mel feature

Answer 20 · 2019-08-26T08:51:49.000Z

@MlWoo i saw that mozilla@554b6df there is silence removal here, and this is needed only to
during training of LPCnet .
Should i remove this code and Train LPCNet.

I have confuse about the trm

@MlWoo i saw that mozilla@554b6df there is silence removal here, and this is needed only to
during training of LPCnet .
Should i remove this code and Train LPCNet.

@alokprasad
I use your LPCTron code, andthe ouput vocie is bad, Is the Hparmas effect?
And by the way, Do we need create mel and linear spectrum in Tacotron preprocess, I don't find we use mel or linear train when we use Tacotron+LPCNet.

Answer 21 · 2019-09-21T02:53:50.000Z

@superhg2012 Is the audio quality of TTS + LPCNet good? how did you make it?

Answer 22 · 2019-10-10T09:53:23.000Z

@byuns9334 I don't get good quality with T2 + LPCNet(20dim). But, I get better quality with T1 and LPCNet(55 dim).

@lmingde I put the dumped f32 files into the audio dir,when train T2, the f32 files in audio dir will be feeded as mel_target for training.

Answer 23 · 2019-10-14T05:00:09.000Z

@superhg2012 i guess T1 and T2 are same except Vocoder part , which anyways we are using LPCNET.
Can you share the changes for T1 with LPCNET.

Answer 24 · 2019-10-14T08:51:42.000Z

@alokprasad about LPCNet model training with 55 dim features is better than 20 dim. About T1, no special changes, just train with 55 dim features.

Answer 25 · 2019-10-14T09:00:07.000Z

nb_features is already 55 , so u mean to say no changes in lpcnet just train lpcnet.
https://github.com/mozilla/LPCNet/blob/master/src/train_lpcnet.py

For T1
Instead of num_mels = 20 you mean num_mels = 55?

Answer 26 · 2019-10-14T09:08:44.000Z

yes, just try it.
when test synthesis... make without taco=1 flag
make clean & make test_lpcnet

Answer 27 · 2019-10-18T19:51:23.000Z

@MlWoo Hi, may I know that, in the training stage when feeding a batch of samples to Tacotron, what padding values are used to ensure the f32 features (whatever 20 or 55 dims) having the same length? I noticed that -0.1 is used in alokprasad's LPCTron implementation.

Answer 28 · 2019-10-19T01:49:02.000Z

@wangfn I have forgotten it. no worries, just mask the padding value when calculating the loss.

Answer 29 · 2019-10-19T09:19:24.000Z

@MlWoo Thanks a lot, indeed masking the padding values is the solution.

Answer 30 · 2020-04-17T03:39:13.000Z

@byuns9334 I don't get good quality with T2 + LPCNet(20dim). But, I get better quality with T1 and LPCNet(55 dim).

@lmingde I put the dumped f32 files into the audio dir,when train T2, the f32 files in audio dir will be feeded as mel_target for training.

@superhg2012 what changes is required for LPCNET for 20 to 55 dim? i thin it uses 55 but only 20 are needed.Any changes in Tactron2 training if we change dims in LPCNET

Answer 31 · 2021-06-15T23:41:36.000Z

Hello all,
I've been working on the tacotron+LPCNet integration but the synthesis is very noisy/robotic. Per the readme I run make dump_data to get the LPCNet training material, then make clean and make dump_data taco=1 and make test_lpcnet taco=1 for the tacotron training and synthesis.

I trained both models on the LJSpeech dataset and ended up with this alignment:

and with these synthesis results
sample.zip

I've heard some great results from others so I am wondering where I went wrong.

Thanks!