Johannes S. Fischer* · Ming Gui* · Pingchuan Ma* · Nick Stracke · Stefan A. Baumann ·Vincent Tao Hu · Björn Ommer
CompVis Group @ LMU Munich
* equal contribution
ECCV 2024 Oral
⇒ code coming soon!
Samples synthesized in
In this work, we leverage the complementary strengths of Diffusion Models (DMs), Flow Matching models (FMs), and Variational AutoEncoders (VAEs): the diversity of stochastic DMs, the speed of FMs in training and inference stages, and the efficiency of a convolutional decoder to map latents into pixel space. This synergy results in a small diffusion model that excels in generating diverse samples at a low resolution. Flow Matching then takes a direct path from this lower-resolution representation to a higher-resolution latent, which is subsequently translated into a high-resolution image by a convolutional decoder. We achieve competitive high-resolution image synthesis at
During training we feed both a low- and a high-res image through the pre-trained encoder to obtain a low- and a high-res latent code. Our model is trained to regress a vector field which forms a probability path from the low- to the high-res latent within
At inference we can take any diffusion model, generate the low-res latent, and then use our Coupling Flow Matching model to synthesize the higher dimensional latent code. Finally, the pre-trained decoder projects the latent code back to pixel space, resulting in
We show zero-shot quantitative comparison of our method against other state-of-the-art methods on the COCO dataset. Our method achieves a good trade-off between performance and computational cost.
We can cascade our models to increase the resolution of a
You can find more qualitative results on our project page.
Please cite our paper:
@misc{fischer2023boosting,
title={Boosting Latent Diffusion with Flow Matching},
author={Johannes S. Fischer and Ming Gui and Pingchuan Ma and Nick Stracke and Stefan A. Baumann and Vincent Tao Hu and Björn Ommer},
year={2023},
eprint={2312.07360},
archivePrefix={arXiv},
primaryClass={cs.CV}
}