/vall-e

An unofficial PyTorch implementation of the audio LM VALL-E, WIP

Primary LanguagePythonMIT LicenseMIT

VALL-E

An unofficial PyTorch implementation of VALL-E, based on the EnCodec tokenizer.

"Buy Me A Coffee"

Install

pip install git+https://github.com/enhuiz/vall-e

Note that the code is only tested under Python 3.10.7.

Usage

  1. Put your data into a folder, e.g. data/your_data. Audio files should be named with the suffix .wav and text files with .normalized.txt.

  2. Quantize the data:

python -m vall_e.emb.qnt data/your_data
  1. Generate phonemes based on the text:
python -m vall_e.emb.g2p data/your_data

  1. Customize your configuration by creating config/your_data/ar.yml and config/your_data/nar.yml. Refer to the example configs in config/test and vall_e/config.py for details. You may choose different model presets, check vall_e/vall_e/__init__.py.

  2. Train the AR or NAR model using the following scripts:

python -m vall_e.train yaml=config/your_data/ar_or_nar.yml

TODO

  • AR model for the first quantizer
  • Audio decoding from tokens
  • NAR model for the rest quantizers
  • Trainers for both models
  • Implement AdaLN for NAR model.
  • Sample-wise quantization level sampling for NAR training.
  • Pre-trained checkpoint and demos on LibriTTS