/Diff-Foley

Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models

Primary LanguagePythonApache License 2.0Apache-2.0

[NeurIPS 2023] Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models

Offical implementation of the NeurIPS 2023 paper: Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models.

Project Page: https://diff-foley.github.io

To-Do:

  • Evaluation Tool ☑️
  • Stage1 CAVP Training Code ☑️
  • Stage2 LDM Training Code ☑️
  • Environment Setting
  • Diff-Foley Inference Code ☑️
  • Diff-Foley Pretrained Model ☑️

News

  • (🔥New) 2023/11/5 Diff-Foley Inference Pipeline is released! See the 'Inference Usages'.
  • (🔥New) 2023/11/5 Diff-Foley Pretrained Model is released! Download from Hugging Face 🤗 here.
  • Including: Stage1-CAVP, Stage2-LDM, Double-Guidance Classifier !!

Inference Usages:

  1. Open the diff_foley_inference.ipynb in inference folder.
  2. Download the pretrained model foler diff_foley_ckpt from Hugging Face 🤗 here and place it under inference folder.
  3. Run the diff_foley_inference.ipynb.

Diff-Foley

BibTeX

@misc{luo2023difffoley, 
title={Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models}, 
author={Simian Luo and Chuanhao Yan and Chenxu Hu and Hang Zhao}, 
year={2023}, 
eprint={2306.17203}, 
archivePrefix={arXiv}, 
primaryClass={cs.SD} 
}