/awesome-talking-head-generation

generate Talking head

awesome-talking-head-generation Awesome

Papers for Talking Head Generation, released codes collections.

This repo mainly focus on the image-driven talking head generation task, but any addition or bug about other domain talking head generation,please open an issue, pull requests or e-mail me by fhongac@cse.ust.hk

Related Group

  1. MMLab@NTU

Datasets

  1. VoxCeleb1 [Download link].
  2. VoxCeleb2 [Download link].
  3. Faceforensics++ [Download link].
  4. CelebV [Download link].
  5. TalkingHead-1KH [Download link].
  6. LRW (Lip Reading in the Wild) [Download link].

Image-driven

2016

  1. [Face2face] Face2face: Real-time face capture and reenactment of RGB videos, CVPR 2016.

2018

  1. [ReenactGAN] ReenactGAN: Learning to Reenact Faces via Boundary Transfer, ECCV 2018. [Code].
  2. [X2Face] X2Face: A network for controlling face generation by using images, audio, and pose codes, ECCV 2018. [Code], [Project].

2019

  1. [FOMM] First order motion model for image animation, NeurIPS 2019. [Code].
  2. [NeuralHead]Few-Shot Adversarial Learning of Realistic Neural Talking Head models, ICCV 2019. [Code].
  3. [Monkey-Net]Animating Arbitrary Objects via Deep Motion Transfer, CVPR 2019 Oral. [Code], [Project].
  4. [fs-vid2vid]Few-shot Video-to-Video Synthesis, NeurIPS 2019. [Code], [Project].

2020

  1. [MeshG] Mesh Guided One-shot Face Reenactment Using Graph Convolutional Networks, ACM Multimedia 2020. [Code].
  2. [MarioNETte] MarioNETte: Few-shot Face Reenactment Preserving Identity of Unseen Targets, AAAI 2020. [Project].

2021

  1. [face-vid2vid] One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing, CVPR 2021 Oral. [Project].

  2. [S2D] Sparse to Dense Motion Transfer for Face Image Animation, ICCV 2021.

  3. [SAFA] SAFA: Structure Aware Face Animation, 3DV 2021. [Code]

  4. [SAA] Self-appearance-aided Differential Evolution for Motion Transfer, arXiv 2021.

  5. [PIRenderer]PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering, ICCV 2021. [Code]

  6. [FaceGAN]FACEGAN: Facial Attribute Controllable rEenactment GAN, WACV 2021.

  7. [F^3A-GAN]F3A-GAN: Facial Flow for Face Animation With Generative Adversarial Networks, IEEE TIP 2021.

2022

  1. [DaGAN]Depth-Aware Generative Adversarial Network for Talking Head Video Generation, CVPR 2022. [Code], [Project]

  2. [TPSM]Thin-Plate Spline Motion Model for Image Animation, CVPR 2022. [Code]

Audio-driven

2016

  1. [LRW] Lip Reading in the Wild, ACCV 2016.

2019

  1. [DAVS] Talking Face Generation by Adversarially Disentangled Audio-Visual Representation, AAAI 2019. [Code].
  2. [ATVGnet] Hierarchical Cross-modal Talking Face Generation with Dynamic Pixel-wise Loss, CVPR 2019. [Code]

2020

  1. [Wav2Lip] A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild, ACM Multimedia 2020. [Code], [Project].
  2. [RhythmicHead]Talking-head Generation with Rhythmic Head Motion, ECCV 2020. [Code].
  3. [MakeItTalk] MakeItTalk: Speaker-Aware Talking-Head Animation, SIGGRAPH Asia 2020. [Code], [Project].

2021

  1. [PC-AVS] Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation, CVPR 2021. [Code], [Project].
  2. [IATS]Imitating Arbitrary Talking Style for Realistic Audio-Driven Talking Face Synthesis,ACM Multimedia 2021..
  3. [EVP] Audio-Driven Emotional Video Portraits, CVPR 2021. [Code]
  4. [FAU] Talking Head Generation with Audio and Speech Related Facial Action Units, arxiv 2021.
  5. [Speech2Talking-Face] Speech2Talking-Face: Inferring and Driving a Face with Synchronized Audio-Visual Representation, IJCAI 2021.
  6. [IATS] Imitating Arbitrary Talking Style for Realistic Audio-Driven Talking Face Synthesis, ACM MM 2021.
  7. [LSP] Live Speech Portraits: Real-Time Photorealistic Talking-Head Animation, ACM TOG 2021.
  8. [Audio2head] Audio2head: Audio-driven one-shot talking-head generation with natural head motion, ArXiv 2021.

2022

  1. [GC-AVT] Expressive Talking Head Generation with Granular Audio-Visual Control , CVPR 2022.

Nerf-Head

2021

  1. [DFA-NeRF] DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Rendering, arxiv, 2021.
  2. [NerFACE] NerFACE: Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction, CVPR 2021 Oral. [Code], [Project]

2022

  1. [SSP-NeRFF] Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation, arxiv, 2022.
  2. [HeadNeRF] HeadNeRF: A Real-time NeRF-based Parametric Head Model, CVPR 2022. [Code], [Project]
  3. [IMavatar] I M Avatar: Implicit Morphable Head Avatars from Videos , CVPR 2022. [Code]
  4. [ROME] Realistic One-shot Mesh-based Head Avatars , ECCV 2022.

Parameter-Based

2020

  1. [DiscoFaceGAN ] Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning , CVPR 2020 Oral. [Code].

Survey

2020

  1. What comprises a good talking-head video generation?: A Survey and Benchmark.