Tacotron for mandarin
An implementation of Tacotron speech synthesis in TensorFlow for mandarin. An sythesized audio sample is given in tmp\ directory.
Background
In April 2017, Google published a paper, Tacotron: Towards End-to-End Speech Synthesis, where they present a neural text-to-speech model that learns to synthesize speech directly from (text, audio) pairs. However, they didn't release their source code or training data. Some time later, Keithito made an independent attempt to provide an open-source implementation of the model described in their paper.
Recently, I forked it from Keithito and trained the model on an opensource mandarin dataset. The result sounds good. Hopefully, it can do better with more effort.
Quick Start
Installing dependencies
-
Install Python 3.
-
Install the latest version of TensorFlow for your platform. For better performance, install with GPU support if it's available. This code works with TensorFlow 1.3 and later.
-
Install requirements: The packages in requirements.txt may not be all necessary, I just freezed my workspace and got tired of cleaning it.
pip install -r requirements.txt
Using a pre-trained model
- Download and unpack a model:
- email:ArwenFeng@foxmail.com
- Run the demo server:
- Note that: When you run the demo server, you should use chinese_cleaners. Remember to confirm it in hparams.py.
After that, you can run the demo server!
cleaners='chinese_cleaners',
python3 demo_server.py --checkpoint tmp/model.ckpt-64000
-
Point your browser at localhost:9000
- Type what you want to synthesize
-
Here are some input samples.
- 今天天气真好
- 孙杨拿了{guan4}军
-
Attention:
- Do not input any punctuations.
- Make sure to change numbers into chinese characters
- When it comes to polyphone problem, you can use pinyin like the example listed above.
Training
Note: you need at least 40GB of free disk space to train a model.
-
Download a speech dataset.
- BZNSYP (from Leo Ma)
-
Unpack the dataset into
~/tacotron_mandarin
After unpacking, your tree should look like this for BZNSYP:
tacotron_mandarin |- BZNSYP |- PhoneLabeling |- ProsodyLabeling |- Wave
-
Preprocess the data
python3 preprocess.py --dataset BZNSYP
-
Train a model Note that: When training BZNSYP dataset, you should use basic_cleaners. Remember to confirm it in hparams.py
cleaners='basic_cleaners',
After adjust hyperparameters, you can train your own models.
python3 train.py
Tunable hyperparameters are found in hparams.py. You can adjust these at the command line using the
--hparams
flag, for example--hparams="batch_size=16,outputs_per_step=2"
. The hyperparameters other than cleaners should generally be set to the same values at both training and eval time.The default hyperparameters are recommended for LJ Speech and other English-language data from Keithito. Although, I didn't change them, they work quite well on this BZNSYP mandarin dataset.
-
Synthesize from a checkpoint
python3 demo_server.py --checkpoint ~/tacotron_mandarin/logs-tacotron/model.ckpt-64000
Replace "64000" with the checkpoint number that you want to use, then open a browser to
localhost:9000
and type what you want to speak. Alternately, you can run eval.py at the command line:python3 eval.py --checkpoint ~/tacotron_mandarin/logs-tacotron/model.ckpt-64000
If you set the
--hparams
flag when training, set the same value here.
Express thanks
Origin Implementation
- By keithito: https://github.com/keithito/tacotron