This repository is the official PyTorch implementation of our AAAI-2022 paper, in which we propose DiffSinger (for Singing-Voice-Synthesis) and DiffSpeech (for Text-to-Speech).
Besides, more detailed & improved code framework, which contains the implementations of FastSpeech 2, DiffSpeech and our NeurIPS-2021 work PortaSpeech is coming soon ✨ ✨ ✨.
DiffSinger/DiffSpeech at training | DiffSinger/DiffSpeech at inference |
---|---|
![]() |
![]() |
🚀 News:
- Dec.01, 2021: DiffSinger was accepted by AAAI-2022.
- Sep.29, 2021: Our recent work
PortaSpeech: Portable and High-Quality Generative Text-to-Speech
was accepted by NeurIPS-2021.
- May.06, 2021: We submitted DiffSinger to Arxiv
.
conda create -n your_env_name python=3.8
source activate your_env_name
pip install -r requirements_2080.txt (GPU 2080Ti, CUDA 10.2)
or pip install -r requirements_3090.txt (GPU 3090, CUDA 11.4)
a) Download and extract the LJ Speech dataset, then create a link to the dataset folder: ln -s /xxx/LJSpeech-1.1/ data/raw/
b) Download and Unzip the ground-truth duration extracted by MFA: tar -xvf mfa_outputs.tar; mv mfa_outputs data/processed/ljspeech/
c) Run the following scripts to pack the dataset for training/inference.
CUDA_VISIBLE_DEVICES=0 python data_gen/tts/bin/binarize.py --config configs/tts/lj/fs2.yaml
# `data/binary/ljspeech` will be generated.
CUDA_VISIBLE_DEVICES=0 python tasks/run.py --config usr/configs/lj_ds_beta6.yaml --exp_name lj_exp1 --reset
CUDA_VISIBLE_DEVICES=0 python tasks/run.py --config usr/configs/lj_ds_beta6.yaml --exp_name lj_exp1 --reset --infer
We also provide:
- the pre-trained model of DiffSpeech;
- the pre-trained model of HifiGAN vocoder;
- the individual pre-trained model of FastSpeech 2 for the shallow diffusion mechanism in DiffSpeech;
Remember to put the pre-trained models in checkpoints
directory.
About the determination of 'k' in shallow diffusion: We recommend the trick introduced in Appendix B. We have already provided the proper 'k' for Ljspeech dataset in the config files.
- WIP. We will provide a form to apply for PopCS dataset.
- WIP. Similar to DiffSpeech.
CUDA_VISIBLE_DEVICES=0 python tasks/run.py --config usr/configs/popcs_ds_beta6.yaml --exp_name popcs_exp1 --reset
# or
CUDA_VISIBLE_DEVICES=0 python tasks/run.py --config usr/configs/popcs_ds_beta6_offline.yaml --exp_name popcs_exp2 --reset
CUDA_VISIBLE_DEVICES=0 python tasks/run.py --config usr/configs/popcs_ds_beta6_offline.yaml --exp_name popcs_exp2 --reset --infer
The pre-trained model for SVS will be provided recently.
tensorboard --logdir_spec exp_name
![]() |
Along vertical axis, DiffSpeech: [0-80]; FastSpeech2: [80-160].
DiffSpeech vs. FastSpeech 2 |
---|
![]() |
![]() |
![]() |
Audio samples can be found in our demo page.
We also put part of the audio samples generated by DiffSpeech+HifiGAN (marked as [P]) and GTmel+HifiGAN (marked as [G]) of test set in resources/demos_1218.
(corresponding to the pre-trained model DiffSpeech)
@misc{liu2021diffsinger,
title={DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism},
author={Jinglin Liu and Chengxi Li and Yi Ren and Feiyang Chen and Zhou Zhao},
year={2021},
eprint={2105.02446},
archivePrefix={arXiv},}
Our codes are based on the following repos:
Also thanks Keon Lee for fast implementation of our work.