Implementation of TediGAN: Text-Guided Diverse Face Image Generation and Manipulation in PyTorch.
Official repository for the paper W. Xia, Y. Yang, J.-H. Xue, and B. Wu. "Text-Guided Diverse Face Image Generation and Manipulation".
Contact: weihaox AT outlook.com
NOTE: The results reported in the paper are about [faces]. We are currently experimenting on other datasets. The codebase includes stylegan training, stylegan inversion, and visual-linguistic learning. The codes will be released when we finish the corresponding training on the new datasets.
[2021/2/16] adding codes for image editing using StyleGAN and CLIP.
We have proposed a novel method (abbreviated as TediGAN) for image synthesis using textual descriptions, which unifies two different tasks (text-guided image generation and manipulation) into the same framework and achieves high accessibility, diversity, controllability, and accurateness for facial image generation and manipulation. Through the proposed multi-modal GAN inversion and large-scale multi-modal dataset, our method can effectively synthesize images with unprecedented quality.
We use the training scripts from genforce. You should prepare the required dataset to train StyleGAN generator (FFHQ for faces or LSUN Bird for birds).
-
Train on FFHQ dataset:
GPUS=8 CONFIG=configs/stylegan_ffhq256.py WORK_DIR=work_dirs/stylegan_ffhq256_train ./scripts/dist_train.sh ${GPUS} ${CONFIG} ${WORK_DIR} -
Train on LSUN Bird dataset:
GPUS=8 CONFIG=configs/stylegan_lsun_bird256.py WORK_DIR=work_dirs/stylegan_lsun_bird256_train ./scripts/dist_train.sh ${GPUS} ${CONFIG} ${WORK_DIR}
Or you can directly use a pretrained StyleGAN generator for ffhq_face_1024, ffhq_face_256, cub_bird_256, or lsun_bird_256.
This step is to find the matching latent codes of given images in the latent space of a pretrained GAN model, e.g. StyleGAN, StyleGAN2, StyleGAN2Ada (should be the same model in the former step). We will include have included the inverted codes in our Multi-Modal-CelebA-HQ Dataset, which are inverted using idinvert.
Our original method is based on idinvert (including StyleGAN training and GAN inversion). To generate 1024 resolution images and show the scalability of our framework, we also learn the visual-linguistic similarity based on pSp.
Due to the scalability of our framework, there are two general ways that can be used to invert a pretrained StyleGAN.
-
Train an image encoder like in idinvert or other GAN inversion methods like pSp or e4e.
-
Project images to latent space directly like in StyleGAN2Ada.
python train_vls.pyWe can also use some powerful pretrained language models, e.g., CLIP, to replace this module. CLIP (Contrastive Language-Image Pre-Training) is a recent a neural network trained on 400 million image and text pairs.
In this case, we have the pretrained image model StyleGAN (or StyleGAN2, StyleGAN2Ada) and the pretrained text encoder CLIP. The inversion process is still necessary. Given the obtained inverted codes of a given image, the desired manipulation or generation result can be simply obtained using the instance-level optimization with an additional CLIP term.
The first step is to install CLIP by running the following commands:
pip install ftfy regex tqdm
pip install git+https://github.com/openai/CLIP.gitThe pretrained model will be downloaded automatically from the OpenAI website (RN50 or ViT-B/32).
The manipulated results can be obtained by simply running:
MODEL_NAME='styleganinv_ffhq256'
IMAGE_LIST='examples/test.list'
python invert_v2.py $MODEL_NAME $IMAGE_LISTParameters:
--loss_weight_clip: weight for the CLIP loss.
--description: a textual description, e.g., he is old.
There is also a text-guided image editing method using CLIP and StyleGAN named StyleClip. Different edits using StyleClip require different values of this parameter. Compared with theirs, our method is not sensitive to the clip weight (cw).
original image, inversion, and manipulated results of clip weight 5/10 with text he is old.
a smiling young woman with short blonde hair
he is young and wears beard
a young woman with long black hair
Below is a curated list of related publications with codes.
- [DF-GAN] Deep Fusion Generative Adversarial Networks for Text-to-Image Synthesis (2020) [paper] [code]
- [ControlGAN] Controllable Text-to-Image Generation (NeurIPS 2019) [paper] [code]
- [DM-GAN] Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis (CVPR 2019) [paper] [code]
- [MirrorGAN] Learning Text-to-image Generation by Redescription (CVPR 2019) [paper] [code]
- [Obj-GAN] Object-driven Text-to-Image Synthesis via Adversarial Training (CVPR 2019) [paper] [code]
- [SD-GAN] Semantics Disentangling for Text-to-Image Generation (CVPR 2019) [paper] [code]
- [HD-GAN] Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Network (CVPR 2018) [paper] [code]
- [AttnGAN] Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks (CVPR 2018) [paper] [code]
- [StackGAN++] Realistic Image Synthesis with Stacked Generative Adversarial Networks (TPAMI 2018) [paper] [code]
- [StackGAN] Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks (ICCV 2017) [paper] [code]
- [GAN-INT-CLS] Generative Adversarial Text to Image Synthesis (ICML 2016) [paper] [code]
- [ManiGAN] ManiGAN: Text-Guided Image Manipulation (CVPR 2020) [paper] [code]
- [Lightweight-Manipulation] Lightweight Generative Adversarial Networks for Text-Guided Image Manipulation (NeurIPS 2020) [paper] [code]
- [SISGAN] Semantic Image Synthesis via Adversarial Learning (ICCV 2017) [paper] [code]
- [TAGAN] Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language (NeurIPS 2018) [paper] [code]
If you find our work, code or the benchmark helpful for your research, please consider to cite:
@article{xia2020tedigan,
title={TediGAN: Text-Guided Diverse Face Image Generation and Manipulation},
author={Xia, Weihao and Yang, Yujiu and Xue, Jing-Hao and Wu, Baoyuan},
journal={arXiv preprint arXiv: 2012.03308},
year={2020}
}