Text-to-Image-Synthesis

Intoduction

This is a pytorch implementation of Generative Adversarial Text-to-Image Synthesis paper, we train a conditional generative adversarial network, conditioned on text descriptions, to generate images that correspond to the description. The network architecture is shown below (Image from [1]). This architecture is based on DCGAN.

Requirements

pytorch
visdom
h5py
PIL
numpy

This implementation currently only support running with GPUs.

Implementation details

This implementation follows the Generative Adversarial Text-to-Image Synthesis paper [1], however it works more on training stablization and preventing mode collapses by implementing:

Feature matching [2]
One sided label smoothing [2]
minibatch discrimination [2] (implemented but not used)
WGAN [3]
WGAN-GP [4] (implemented but not used)
SRCNN
GAN-BERT (implemented but not integrated)

Datasets

We used Flowers datasets, we converted each dataset (images, text embeddings) to hd5 format.

We used the text embeddings provided by the paper authors

To use this code you can either:

Use the converted hd5 datasets, flowers
Convert the data youself
1. download the dataset as described here
2. Add the paths to the dataset to config.yaml file.
3. Use convert_flowers_to_hd5_script script to convert the dataset.

Hd5 file taxonomy `

split (train | valid | test )
- example_name
  - 'name'
  - 'img'
  - 'embeddings'
  - 'class'
  - 'txt'

Usage

Training

python runtime.py

Arguments:

type : GAN archiecture to use (gan | wgan | vanilla_gan | vanilla_wgan). default = gan. Vanilla mean not conditional
dataset: Dataset to use (birds | flowers). default = flowers
split : An integer indicating which split to use (0 : train | 1: valid | 2: test). default = 0
lr : The learning rate. default = 0.0002
diter : Only for WGAN, number of iteration for discriminator for each iteration of the generator. default = 5
vis_screen : The visdom env name for visualization. default = gan
save_path : Path for saving the models.
l1_coef : L1 loss coefficient in the generator loss fucntion for gan and vanilla_gan. default=50
l2_coef : Feature matching coefficient in the generator loss fucntion for gan and vanilla_gan. default=100
pre_trained_disc : Discriminator pre-tranined model path used for intializing training.
pre_trained_gen Generator pre-tranined model path used for intializing training.
batch_size: Batch size. default= 64
num_workers: Number of dataloader workers used for fetching data. default = 8
epochs : Number of training epochs. default=200
cls: Boolean flag to whether train with cls algorithms or not. default=False

Image Super Resolution

We have used SRCNN (Super-Resolution Convolutional Neural Network) and Bicubic interpolation inorder to enhance the generated image.

Command:

cd SRCNN

cd SRCNN-pytorch

python test.py --weights-file "BLAH_BLAH/srcnn_x3.pth" --image-file "data/a flower that has violet petals that are surrounding a cluster of stamen_.jpg” —scale 4

python test.py --weights-file "BLAH_BLAH/srcnn_x3.pth" --image-file "data/this flower is white and trumpet shaped with yellow-green lines running from the center of the flowe.jpg" --scale 4