/DF-GAN

Deep Fusion Generative Adversarial Networks for Text-to-Image Synthesis

Primary LanguagePython

DF-GAN: Deep Fusion Generative Adversarial Networks for Text-to-Image Synthesis

(A novel and effective one-stage Text-to-Image Backbone)

Official Pytorch implementation for our paper DF-GAN: Deep Fusion Generative Adversarial Networks for Text-to-Image Synthesis by Ming Tao, Hao Tang, Songsong Wu, Nicu Sebe, Xiaoyuan Jing, Fei Wu, Bingkun Bao.

Requirements

  • python 3.6+
  • Pytorch 1.0+
  • easydict
  • nltk
  • scikit-image
  • A titan xp (set nf=32 in *.yaml) or a V100 32GB (set nf=64 in *.yaml)

Installation

Clone this repo.

git clone https://github.com/tobran/DF-GAN
cd DF-GAN/code/

Datasets Preparation

  1. Download the preprocessed metadata for birds coco and save them to data/
  2. Download the birds image data. Extract them to data/birds/
  3. Download coco dataset and extract the images to data/coco/

Pre-trained text encoder

  1. Download the pre-trained text encoder for CUB and save it to DAMSMencoders/bird/inception/
  2. Download the pre-trained text encoder for coco and save it to DAMSMencoders/coco/inception/

Training

Train DF-GAN models:

  • For bird dataset: python main.py --cfg cfg/bird.yml

  • For coco dataset: python main.py --cfg cfg/coco.yml

  • *.yml files are example configuration files for training/evaluation our models.

Evaluating

Dwonload Pretrained Model

Evaluate DF-GAN models:

  • To evaluate our DF-GAN on CUB, change B_VALIDATION to True in the bird.yml. and then run python main.py --cfg cfg/bird.yml
  • To evaluate our DF-GAN on coco, change B_VALIDATION to True in the coco.yml. and then run python main.py --cfg cfg/coco.yml
  • We compute inception score for models trained on birds using StackGAN-inception-model.
  • We compute FID for CUB and coco using DM-GAN/eval/FID.

Citing DF-GAN

If you find DF-GAN useful in your research, please consider citing our paper:

@article{ming2020DFGAN,
  title={DF-GAN: Deep Fusion Generative Adversarial Networks for Text-to-Image Synthesis},
  author={Ming Tao, Hao Tang, Songsong Wu, Nicu Sebe, Fei Wu, Xiao-Yuan Jing},
  journal={arXiv preprint arXiv:2008.05865},
  year={2020}
}

The code is released for academic research use only. For commercial use, please contact Ming Tao.

Reference