CorentinJ/Real-Time-Voice-Cloning

I trained a German model - which files do I need to share?

padmalcom opened this issue · 5 comments

padmalcom commented

Hi I trained encoder, synthesizer and vocoder on German female voices from m-ailabs and even if results are not too good, I'd like to share it. Encoder and vocoder folders aren't too big, but the synthesizer folder is ~10gb. Does anybody know if I can leave some files out or do I have to share the entire 10 gb folder?

Deleted user commented

Hi @padmalcom , congratulations on completing full stack training and thank you for sharing your German models with the community! For the pretrained models, this is the minimum set of files we distribute. If this is not enough guidance on which files should be selected, please ask.

encoder\saved_models\pretrained.pt
synthesizer\saved_models\logs-pretrained\taco_pretrained\checkpoint
synthesizer\saved_models\logs-pretrained\taco_pretrained\tacotron_model.ckpt-278000.data-00000-of-00001
synthesizer\saved_models\logs-pretrained\taco_pretrained\tacotron_model.ckpt-278000.index
synthesizer\saved_models\logs-pretrained\taco_pretrained\tacotron_model.ckpt-278000.meta
vocoder\saved_models\pretrained\pretrained.pt

If you modified any of the following files, please also include them:

encoder\params_model.py
encoder\params_data.py
synthesizer\hparams.py
synthesizer\utils\cleaners.py
synthesizer\utils\symbols.py
synthesizer\utils\text.py
vocoder\hparams.py
padmalcom commented

Thanks, filesize is now 372mb which is fantastic. Here are the most important points I learned during training. Will upload the archive soon.

First things first

  • Trained on female German voices of m-ailabs
  • Tests show that the vocoder training is not neccessary, so focus on encoder and synthesizer
  • I had to edit some parts of the implementation to make it work with m-ailabs
  • m-ailabs requires a lot of cleaning (at least for German voices). I added a script to do most of the work.
    • mailab_normalize_text.py: Creates text files besides each wav file of the m-ailabs dataset which apparently is required.

How to run?

Please note that all changes I did were made for windows. You might want to adapt it for linux.

Download

If you just want to apply the model, download the bin.zip from the following release and unzip it to your repository.
https://github.com/padmalcom/Real-Time-Voice-Cloning-German/releases/tag/0.1

Training

To train the model replace all code files in the repositlry by the code files from my release. You can leave out the bin.zip file from the release.

Encoder

  • python encoder_preprocess.py E:\Datasets\
  • python encoder_train.py my_run E:\Datasets\SV2TTS\encoder

Synthesizer

  • Make sure you updated synthesizer/utils/symbols.py for your language
  • python synthesizer_preprocess_audio.py E:\Datasets\ --subfolders de_DE\by_book\female\ --dataset "" --no_alignments --wav_dir
  • python synthesizer_preprocess_embeds.py E:\Datasets\SV2TTS\synthesizer
  • python synthesizer_train.py my_run E:\Datasets\SV2TTS\synthesizer

Vocoder (You really do not need to train the vocoder, it works well as the pretrained model is - at least for German)

  • pip install librosa==0.7.2
  • python vocoder_preprocess.py E:\Datasets\ --model_dir=synthesizer/saved_models/logs-my_run/
  • python vocoder_train.py my_run E:\Datasets\

Toolbox

  • pip install matplotlib==3.2.2 #required for toolbox to work

Todos and Learnings

  • Getting the Toolbox to run with a specific voice of the four voices does not seem to work properly. I still have to figure out why (not).
  • Next experiment will be to merge all voices into one voice package and see if the result improves.
ALIXGUSTAF commented

@padmalcom please can you guide me how is adding Arabic language to [Real-Time-Voice-Cloning] program step by step

joachimschwarzschild commented

@padmalcom i want to clone a german voice for my instagram reels
can you make an video how to install it on windows?

  • 👍1
  • ❤️1