/AvatarMe

Public repository for the CVPR paper AvatarMe

Youtube Video arXiv Prepring

Public repository for the CVPR 2020 paper AvatarMe, with high resolution results, data and more.

Alexandros Lattas 1,2, Stylianos Moschoglou 1,2, Baris Gecer 1,2, Stylianos Ploumpis 1,2, Vasileios Triantafyllou 2,
Abhijeet Ghosh 1, Stefanos Zafeiriou 1,2
1 Imperial College London
2 FaceSoft

[Paper PDF] [CVPR 2020]

Overview

Intro Image

AvatarMe is the first method that is able to reconstruct photorealistic 3D faces from a single ‘in-the-wild” image with an increasing level of detail. To achieve this, we capture a large dataset of facial shape and reflectance and build on a state-of-the 3D texture and shape reconstruction method and successively refine its results in order to generate the high-resolution diffuse and specular components that are required for realistic rendering.

Method

Method Image

A 3DMM is fitted to an ''in-the-wild'' input image and a completed UV texture is synthesized, while optimizing for the identity match between the rendering and the input. The texture is up-sampled 8 times, to synthesize plausible high-frequency details. We then use an image translation network to de-light the texture and obtain the diffuse albedo with high-frequency details. Then, separate networks are used to infer the specular albedo, diffuse normals and specular normals from the diffuse albedo and the 3DMM shape normals. Moreover, the networks are trained on 512x512 patches and inferences are ran on 1536x1536 patches with a sliding window. Finally, we can transfer the facial shape and the consistently inferred reflectance to a head model. Both face and head can be rendered realistically in any environment.

Patch process for Diffuse Albedo Patch prosses for Specular Albedo

Patch process for Diffuse and Specular Abedo

Patch process for Diffuse Normals Patch process for Specular Normals

Patch process for Diffuse and Specular Normals

Results

For high quality rendered results see the following youtube video.

Click for HD video results Click for HD video results

Data

Information about the public training data will be anounced as soon as possible.

Citation

If you find this work useful, please use the following to cite our paper:

@article{lattas2020avatarme,
  title={AvatarMe: Realistically Renderable 3D Facial Reconstruction "in-the-wild"},
  author={Lattas, Alexandros and Moschoglou, Stylianos and Gecer, Baris and Ploumpis, Stylianos and Triantafyllou, Vasileios and Ghosh, Abhijeet and Zafeiriou, Stefanos},
  journal={arXiv preprint arXiv:2003.13845},
  year={2020}
}