MTGAN

MT GAN

python 2.7

Pytorch

In addition, please add the project folder to PYTHONPATH and pip install the following packages:

Data

Training

Pre-train DAMSM models:
- For bird dataset: python3 pretrain_DAMSM.py --cfg cfg/DAMSM/bird.yml --gpu 0
- For coco dataset: python3 pretrain_DAMSM.py --cfg cfg/DAMSM/coco.yml --gpu 0
Train AttnGAN models:
- For bird dataset: python3 main.py --cfg cfg/bird_attn2.yml --gpu 0
- For coco dataset: python3 main.py --cfg cfg/coco_attn2.yml --gpu 0
*.yml files are example configuration files for training/evaluation our models.

Pretrained Model

DAMSM for bird. Download and save it to DAMSMencoders/
DAMSM for coco. Download and save it to DAMSMencoders/
AttnGAN for bird. Download and save it to models/
AttnGAN for coco. Download and save it to models/
AttnDCGAN for bird. Download and save it to models/
- This is an variant of AttnGAN which applies the propsoed attention mechanisms to DCGAN framework.

Sampling

Run python3 main.py --cfg cfg/eval_bird.yml --gpu 0 to generate examples from captions in files listed in "./data/birds/example_filenames.txt". Results are saved to DAMSMencoders/.
Change the eval_*.yml files to generate images from other pre-trained models.
Input your own sentence in "./data/birds/example_captions.txt" if you wannt to generate images from customized sentences.

Validation

To generate images for all captions in the validation dataset, change B_VALIDATION to True in the eval_*.yml. and then run python main.py --cfg cfg/eval_bird.yml --gpu 1
We compute inception score for models trained on birds using StackGAN-inception-model.
We compute inception score for models trained on coco using improved-gan/inception_score.

Examples generated by AttnGAN [Blog]

Evaluation code embedded into a callable containerized API is included in the eval\ folder.

Reference

ErSKS/MTGAN