A (Heavily Documented) TensorFlow Implementation of Tacotron: A Fully End-to-End Text-To-Speech Synthesis Model

Major History

June 21, 2017. Fourth draft.
- I've updated the code for TF 1.1 to TF 1.2. Turns out that TF 1.2 has a new api for attention wrapper and more detailed options.
- I've added a sanity check option to the hyperparams.py file. Basically, it's purpose is to find out if our model is able to learn a handful of training data wihtout caring about generalization. For that, the training was done on a single mini-batch (32 samples) over and over again, and sample generation was based on the same text. I observed a quite smooth training curve for as below, and after around 18K global steps it started to generate recognizable sounds. The sample results after 36K steps are available in the logdir_s folder. It took around seven hours on a single gtx 1080. The pretrained files can be downloaded from here. The training curve looks like this.

June 4, 2017. Third draft.
- Some people reported they gained promising results, based on my code. Among them are, @ggsonic, @chief7. To check relevant discussions, see this discussion, or their repo.
- According @ggsonic, instance normalization worked better than batch normalization.
- @chief7 trained on pavoque data, a German corpus spoken by a single male actor. He said that instance normalization and zero-masking are good choices.
- Yuxuan, the first author of the paer, advised me to do sanity-check first with small data, and to adjust hyperparemters since our dataset is different from his. I really appreciate his tips, and hope this would help you.
- Alex's repo, which is another implementation of Tacotron, seems to be successful in getting promising results with some small dataset. He's working on a big one.
June 2, 2017.
- Added train_multiple_gpus.py for multiple GPUs.
June 1, 2017. Second draft.
- I corrected some mistakes with the help of several contributors (THANKS!), and re-factored source codes so that they are more readable and modular. So far, I couldn't get any promising results.
May 17, 2017. First draft.
- You can run it following the steps below, but good results are not guaranteed. I'll be working on debugging this weekend. (Code reviews and/or contributions are more than welcome!)

Requirements

NumPy >= 1.11.1
TensorFlow == 1.2
librosa
tqdm

Data

Since the original paper was based on their internal data, I use a freely available one, instead.

The World English Bible is a public domain update of the American Standard Version of 1901 into modern English. Its text and audio recordings are freely available here. Unfortunately, however, each of the audio files matches a chapter, not a verse, so is too long for many machine learning tasks. I had someone slice them by verse manually. You can download the audio data and its text from my dropbox.

File description

hyperparams.py includes all hyper parameters that are needed.
prepare_pavoque.py creates sliced sound files from raw sound data, and constructs necessary information.
prepro.py loads vocabulary, training/evaluation data.
data_load.py loads data and put them in queues so multiple mini-bach data are generated in parallel.
utils.py has several custom operational functions.
modules.py contains building blocks for encoding/decoding networks.
networks.py has three core networks, that is, encoding, decoding, and postprocessing network.
train.py is for training.
eval.py is for sample synthesis.

Training

STEP 1. Adjust hyper parameters in hyperparams.py if necessary.
STEP 2. Download and extract the audio data and its text.
STEP 3. Run train.py. or train_multi_gpus.py if you have more than one gpu.

Sample Synthesis

Run eval.py to get samples.

Acknowledgements

I would like to show my respect to Dave, the host of www.audiotreasure.com and the reader of the audio files.

ZVK/tacotron