- [30/03/2024]: The evaluation code is updated.
- [07/02/2024]: The inference script is released.
- [06/02/2024]: The model weight is released.
conda create --name diffspeaker python=3.9
conda activate diffspeaker
Install MPI-IS. Follow the command in MPI-IS to install the package. Depending on if you have /usr/include/boost/ directories, The command is likely to be
git clone https://github.com/MPI-IS/mesh.git
cd mesh
sudo apt-get install libboost-dev
python -m pip install pip==20.2.4
BOOST_INCLUDE_DIRS=/usr/include/boost/ make all
python -m pip install --upgrade pip
Then install the rest of the dependencies.
cd ..
git clone https://github.com/theEricMa/DiffSpeaker.git
cd DiffSpeaker
pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113
pip install imageio-ffmpeg
pip install -r requirements.txt
You can access the model parameters by clicking here. Place the checkpoints folder into the root directory of your project. This folder includes the models that have been trained on the BIWI and vocaset datasets, utilizing wav2vec and hubert as the backbones.
For the BIWI model, use the script below to perform inference on your chosen audio files. Specify the audio file using the --example argument.
sh scripts/demo/demo_biwi.sh
For the vocaset model, run the following script.
sh scripts/demo/demo_vocaset.sh
To obtain the metrics reported in the paper, use the scripts in scripts/diffusion/biwi_evaluation and scripts/diffusion/vocaset_evaluation. For example, to evaluate DiffSpeaker in BIWI dataset with the hubert backbone, use the following script.
sh scripts/diffusion/biwi_evaluation/diffspeaker_hubert_biwi.sh
mkdir experiments