/fewshot-face-translation-GAN

Generative adversarial networks integrating modules from FUNIT and SPADE for face-swapping.

Primary LanguageJupyter Notebook

Few-shot face translation Source face: Mona Lisa

A GAN based approach for one model to swap them all.

The table below shows our priliminary face-swapping results requiring one source face and <=5 target face photos. Notice that almost all of the identities, except Stephen Curry, are not in our training data (which is a subset of VGGFace2). More translation results can be found here.

Also, our model is capable of producing faces that has its gaze direction, glasses, and hiar occlusions being consistent with given source face. However, our model has suboptimal performance in terms of translating to asian faces. This is possibly due to limited representability of the feature extractor.

Src.\Tar. Andrej Karpathy Andrew Y. Ng Du Fu Elon Musk Emilia Clarke Geoffrey Hinton Stephen Curry Yann Lecun Yoshua Benjio
Andrej
Karpathy		 N/A
Andrew
Y. Ng		 N/A
Du Fu N/A
Elon
Musk		 N/A
Emilia
Clarke		 N/A
Geoffrey
Hinton		 N/A
Stephen
Curry		 N/A
Yann
Lecun		 N/A
Yoshua
Benjio		 N/A

I really like the Du Fu translation: such an interesting demostration how the GAN imagine the appearance of the prominent Chinese poet from just a painting.

Try in Google Colab

  • master branch (Jun. 2019) Open In Colab
  • dev branch (Oct. 2019) Open In Colab
We only provide pre-trained weights and inference script for now. Training script will be released after code cleanup.

Architecture

The above image illustrates our generator, which is a encoder-decoder based network, at test phase. Our swap-them-all approach is basically a GAN conditioned on the latent embeddings extracted from a pre-trained face recognition model. SPADE and AdaIN modules are incorporated in order to inject semantic priors to the networks.

During training phase, the input face A is heavily blurred and we train the model with resonctruction loss. Other objectives that aimed to improve translation performance while keeping semantic consistency, such as perceptual loss on rgb output and cosine similarity loss on latent embeddings, are also introduced.

Things that didn't work

  1. We tried to distort (spline warp, downsample) the input image as in faceswap-GAN instead of masking it. However, the model did not learn proper identity translation but output face that is similar to its input.

Requirements

  • Python 3.6
  • Keras 2.2.4
  • TensorFlow 1.12.0 or 1.13.1

References

  1. Semantic Image Synthesis with Spatially-Adaptive Normalization
  2. Few-Shot Unsupervised Image-to-Image Translation
  3. DEEP LEARNING FOR FASHION AND FORENSICS