Diffusion models have obtained substantial progress in image-to-video (I2V) generation. However, such models are not fully understood. In this paper, we report a significant but previously overlooked issue in I2V diffusion models (I2V-DMs), namely, conditional image leakage. I2V-DMs tend to over-rely on the conditional image at large time steps, neglecting the crucial task of predicting the clean video from noisy inputs, which results in videos lacking dynamic and vivid motion. We further address this challenge from both inference and training aspects by presenting plug-and-play strategies accordingly. These strategies are validated on various I2V-DMs including DynamiCrafter, SVD and VideoCrafter1.
- [2024.06.25]: Release code, models and project page.
- Release the models without watermarks and in other resolutions.
- Applying our strategy on Vidu, a transformer-based text-to-video diffusion model.
Our plug-and-play strategies can be applied to various I2V-DMs, allowing the direct use of their original environments. For example, to set up DynamiCrafter:
cd examples/DynamiCrafter
conda create -n dynamicrafter python=3.8.5
conda activate dynamicrafter
pip install -r requirements.txt
To set up VideoCrafter1:
cd examples/VideoCrafter
conda create -n videocrafter python=3.8.5
conda activate videocrafter
pip install -r requirements.txt
To set up SVD:
cd examples/SVD
conda create -n svd python=3.9.18
conda activate svd
pip install -r requirements.txt
Download the WebVid dataset from here, where we use Webvid-2M subset. Put .csv file in examples/dataset/results_2M_train.csv and video data in examples/dataset/. We use the raw data without any filters.
We clone the repo of DynamiCrafter and VideoCrafter1 and implement SVD by ourselves. We apply our plug and plug-and-play strategies on them.
| Model | Resolution | Initial Noise |
|---|---|---|
| DynamiCrafter | 256x256 | Initial Noise |
| DynamiCrafter | 320x512 | Initial Noise |
| DynamiCrafter | 576x1024 | Initial Noise |
| VideoCrafter | 256x256 | Initial Noise |
| VideoCrafter | 320x512 | Initial Noise |
| VideoCrafter | 576x1024 | Initial Noise |
| SVD | 320 x 512 | Initial Noise |
| SVD | 576 x 1024 | Initial Noise |
| Model | Conditional image | Standard inference | + Our inference strategy |
| DynamiCrafter320x512 |
|
|
|
| VideoCrafter320x512 |
|
|
|
| SVD 576x1024 |
|
|
|
- Download the original DynamiCrafter checkpoints from the repository and put it in
examples/DynamiCrafter/ckpt/original,or download our DynamiCrafter-CIL from here and put it inexamples/DynamiCrafter/ckpt/finetuned. Download the initial noise in the above table and put it inexamples/DynamiCrafter/ckpt/. - Run the following commands:
cd examples/DynamiCrafter
# for original DynamiCrafter with 320x512 resolution
sh inference_512.sh
# for our DynamiCrafter-CIL with 320x512 resolution
sh inference_CIL_512.sh
# for our DynamiCrafter-CIL with 576x1024 resolution
sh inference_CIL_1024.sh
The relevant parameters in inference.sh for our strategy are explained as follows:
M: the start timestep M.whether_analytic_init: indicates whether to use Analytic-Init; 0 means it is not applied, while 1 means it is appliedanalytic_init_path: the path for initializing the mean and variance of the noise if Analytic-Init is applied
Note that M=1000, whether_analytic_init=0 is the baseline.
The effect of start time M is as follows:
| Conditional image | M=1.00T | M=0.94T | M=0.90T | M=0.86T | M=0.82T |
|
|
|
|
|
|
- Download the pretrained SVD model and put it in
examples/SVD/ckpt/pretrained/stable-video-diffusion-img2vid. Download our SVD-CIL from here and put it inexamples/SVD/ckpt/finetuned. Download the initial noise in the above table and put them inexamples/SVD/ckpt/. - Run the following commands:
cd examples/SVD
# for original SVD
sh inference.sh
# for SVD-CIL with 320x512 resolution
sh inference_CIL_512.sh
The relevant parameters for inference are set in examples/SVD/config/inference.yaml,which are explained as follows:
sigma_max: the start time M.analytic_init_path: the path for initializing the mean and variance of the noise if Analytic-Init is applied
- Download the original VideoCrafter checkpoints from the repository and put it in
examples/VideoCrafter/ckpt/original,or download our VideoCrafter-CIL from here and put it inexamples/VideoCrafter/ckpt/finetuned.Download the initial noise in the above table and put them inexamples/VideoCrafter/ckpt. - Run the following commands:
cd examples/VideoCrafter
# for original VideoCrafter with 320x512 resolution
sh inference_512.sh
# for VideoCrafter-CIL with 320x512 resolution
sh inference_CIL_512.sh
The relevant parameters in inference.sh for our strategy are explained as follows:
M: the start time Manalytic_init_path: the path for initializing the mean and variance of the noise if Analytic-Init is applied
Similar to the inference strategy, we finetune the baselines based on the repository DynamiCrafter, VideoCrafter1 and SVD.
| Model | Conditional image | Finetuned-Baseline | + Our training strategy |
| DynamiCrafter |
|
|
|
| VideoCrafter |
|
|
|
| SVD |
|
|
|
- Download the DynamiCrafter checkpoints from the repository and put them in
examples/DynamiCrafter/ckpt/original. - Run the following commands:
cd examples/DynamiCrafter
sh train.sh
The relevant parameters in train.sh for our strategy are explained as follows:
-
beta_m: the maximum noise level. -
a: the exponent of center of the distribution:$\mu(t)=2t^a-1$ , where$a > 0$ .
The effect of beta_mis as follows:
| Conditional Image | beta_m=25 | beta_m=100 | beta_m=700 |
|
|
|
|
The effect of a is as follows:
| Conditional Image | a = 5.0 | a = 1.0 | a = 0.1 |
|
|
|
|
|
Lower a correspond to more dynamic motion and lower temporal consistency and image alignment.
- Download the SVD checkpoints from the repository and put them in
examples/SVD/ckpt/pretrained/stable-video-diffusion-img2vid; - Run the following commands:
cd examples/SVD
sh train.sh
The relevant parameters in examples/SVD/config/train.yamlfor our strategy are explained as follows:
-
beta_m: the maximum noise level. Higher beta_m correspond to more dynamic motion and lower temporal consistency and image alignment. -
a: the exponent of center of the distribution:$\mu(t)=2t^a-1$ , where$a > 0$ . Lower$a$ correspond to more dynamic motion and lower temporal consistency and image alignment.
Note that original SVD first add noise on conditional image and then feed it into VAE. Here we first feed the conditional image into VAE and then add noise on the conditional latents.
- Download the VideoCrafter checkpoints from the repository and put them in
examples/VideoCrafter/original/ckpt/. - Run the following commands:
cd examples/VideoCrafter
sh train.sh
The relevant parameters in train.sh for our strategy are explained as follows:
-
beta_m: the maximum noise level. Higher beta_m correspond to more dynamic motion and lower temporal consistency and image alignment. -
a: the exponent of center of the distribution:$\mu(t)=2t^a-1$ , where$a > 0$ . Lower$a$ correspond to more dynamic motion and lower temporal consistency and image alignment.
Naive fine-tuning and ours were trained under the same settings for fair comparison. In the future, we will release the model without watermark.
| Model | Naive Fine-Tuning | Ours on Webvid | Watermark-free |
|---|---|---|---|
| DynamiCrafter | 320x512 | 320x512 | 320x512 576x1024 |
| SVD | 320x512 | 320x512 | |
| VideoCrafter1 | 320x512 | 320x512 |
If you find this repository helpful, please cite as:
@article{zhao2024identifying,
title={Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model},
author={Zhao, Min and Zhu, Hongzhou and Xiang, Chendong and Zheng, Kaiwen and Li, Chongxuan and Zhu, Jun},
journal={arXiv preprint arXiv:2406.15735},
year={2024}
}This implementation is based on the following work:
- DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors
- Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
- VideoCrafter1: Open Diffusion Models for High-Quality Video Generation Thanks to the authors for sharing their code and models.