/ganimation_replicate

An Out-of-the-Box Replication of GANimation using PyTorch, pretrained weights are available!

Primary LanguagePythonMIT LicenseMIT

GANimation -- An Out-of-the-Box Replicate

Status Platform PyTorch License

A reimplementation of GANimation: Anatomically-aware Facial Animation from a Single Image, using PyTorch. Pretrained models/weights are available at GDrive or BaiduPan(Code:3fyb) !

ganimation_show

Pros (compared with the official implementation)

  • Codes are cleaner and well structured, inspired by the pytorch-CycleGAN-and-pix2pix.
  • Provide a more powerful test function for generating linear interpolations between two expressions as shown in the paper.
  • Provide a preprocessed CelebA dataset, including cropped faces, Action Units related to all cropped faces, train and test split.
  • Provide pretrained models for the above CelebA dataset (trained with ~145k images for 30 epoches).
  • Provide Action Units vectors for the EmotionNet extracted using OpenFace.
  • Provide pretrained models for the EmotionNet dataset (trained with ~410k images for 30 epoches).

All resources related to this project are located at GDrive or BaiduPan(Code:3fyb).

Getting Started

Requirements

  • Python 3
  • PyTorch 0.4.1
  • visdom (optional, only for training with browser visualizer)
  • imageio (optional, only for generating GIF image in testing)

Installation

  • Clone this repo:
git clone https://github.com/donydchen/ganimation_replicate.git
cd ganimation_replicate
pip install -r requirements.txt

Resources

  • All resources related to this project are located at GDrive or BaiduPan(Code:3fyb).
  • Download datasets and put it in the root path of this project.
  • Download ckpts and put it in the root path of this project. (optional, only for test or finetune)
  • Note: for the EmotionNet, the AU vectors are saved as a dictionary, where the key is the file name (without extension), and dumped into a pickle file.

Train

  • To view training results and loss plots, run python -m visdom.server and click the URL http://localhost:8097
python main.py --data_root [path_to_dataset]

# e.g. python main.py --data_root datasets/celebA --gpu_ids 0,1 --sample_img_freq 500
#      python main.py --data_root datasets/emotionNet --gpu_ids 0,1 --sample_img_freq 500
#      set '--visdom_display_id 0' if you don't want to use visdom
#      use 'python main.py -h' to check out more options.

Test

  • Make sure you have trained the model or downloaded the pretrained model.
python main.py --mode test --data_root [path_to_dataset] --ckpt_dir [path_to_pretrained_model] --load_epoch [epoch_num]

# e.g. python main.py --mode test --data_root datasets/celebA --batch_size 8 --max_dataset_size 150 --gpu_ids 0,1 --ckpt_dir ckpts/celebA/ganimation/190327_161852/ --load_epoch 30
#      set '--interpolate_len 1' if you don't need linear interpolation.
#      use '--save_test_gif' to generate animated images.

Finetune

python main.py --data_root [path_to_dataset] --ckpt_dir [path_to_existing_checkpoint] --load_epoch [epoch_num] 

# e.g. python main.py --data_root datasets/celebA --gpu_ids 0,1 --sample_img_freq 300 --n_threads 18 --ckpt_dir ckpts/celebA/ganimation/190327_161852 --load_epoch 30 --epoch_count 31 --niter 30 --niter_decay 10

Use Own Datasets

  • Crop Face: Use face_recognition to extract face bounding box and crop face from images.
  • Obtain AUs Vector: Use OpenFace to extract Action Units vectors from the above cropped face. Specifically, only the AUs intensity is used in this project, namely AU01_r, AU02_r, AU04_r, AU05_r, AU06_r, AU07_r, AU09_r, AU10_r, AU12_r, AU14_r, AU15_r, AU17_r, AU20_r, AU23_r, AU25_r, AU26_r, AU45_r.
./FaceLandmarkImg -f [path_to_img] -aus

# In the result file, values of columns [2:19] are extracted for later usage.
  • Download Pretrained Model: Since in this project, the EmotionNet employed for training contains more than 400k in-the-wild face images, the pretrained model should meet the requirements of lots of scenes. You're recommended to directly try to apply the EmotionNet pretrained model on your own datasets.

Some Results

CelebA

Training

celeba_training

Testing (with GANimation model, on epoch 30)

celeba_testing

Testing (with StarGAN model, on epoch 30)

celeba_stargan_testing

EmotionNet （Visual quality is much better than that of CelebA）

Training

emotionnet_training

Testing (with GANimation model, on epoch 30)

emotionnet_testing

Testing (with StarGAN model, on epoch 30)

emotionnet_stargan_testing

Why this Project?

My mentor came up with a fancy idea of playing GANs with AUs when I was an intern at AI Lab, Lenovo Research around early August, 2018. I enjoyed the idea very much and started working on it. However, just a few days after that, the GANimation paper showed up, which was not a good news for us... So I tried to replicate GANimation, and this is the start of this project.

And in late August, 2018, I came accross an issue on the official GANimation implementation, claiming that the test result is wrong. While in my case, I did get some reasonable results, so I replied that issue with the results I had got. Since the author of GANimation hadn't decided to release the pretrained model yet, I recieved Emails inquiring me whether I could offer my codes and pretrained models from time to time.

I really wanted to provide the codes and pretrained models. However, I was very busy in the past few months, moving from Beijing to Singapore, working for paper deadlines, so on and so forth. So the codes remained in the server of Lenovo Research for half an year. And these days, I finally got some free time. So I dug out the codes, cleaned them, retrained the network, and now, I make them public. I will keep updating this project if I have time, and hope that these codes can serve to faciliate the research of someone who are working on the related tasks.

Feel free to contact me if you need any help from me related to this project.

Pull Request

You are always welcome to contribute to this repository by sending a pull request.