This is the repository for the master thesis project of a 3D virtual chatbot. The repository consists of various google colab notebooks that can help you kickstart the project. Specifically, it supports a quick plug and play of various models, that are crucial for understanding the state-of-the-art and implementing novel solutions for 3D avatar reconstruction, inverse rendering based on a single image, synthetic image generation (GAN based) for expression transfer, 3D avatar animation along with voice synchronization :
Dwelve into the world of 3D virtual chatbots, by trying the following Google Colab notebooks
 |
 |
|---|---|
| Synthetic image generation - StyleGAN | Photorealistic video-to-video inference (few-shot) |
 |
 |
|---|---|
| Reconstruct 3D FLAME mesh from single image | Expression Transfer from source image to target mesh |
 |
 |
|---|---|
| Fit 3D avatar reconstruction pipeline | 3D Avatar animation based on audio signal - VOCA |
 |
 |
|---|---|
| Text-to-audio synthesis | Inverse rendering for input video (NHA) |
 |
 |
|---|---|
| Visualize Mesh | 3D face mesh reconstruction from single-view image |
Our tutorials include code from the following models/papers/repositories:
- fs_vid2vid → fs_vid2vid_inference.ipynb
- VOCA → voca.ipynb
- DECA → deca.ipynb , DECAthlon.ipynb
- StyleGAN3 → stylegan.ipynb
- unsup3D → unsup3d.ipynb
- NHA - Neural Head Avatars → neural_head_avatars.ipynb
The notebooks allow also, customized testing of the models. This means that any examples of face images or audio sequences can be used for inference or testing. The novel implementation of an end-to-end Detailed Textured Avatar Enhancer named DECAthlon is also included in the repository.
The Google Colab notebooks have detailed explanations on how ot install and run all models. Please refer to the respective notebooks, or follow the guidelines for a clean local installation (Linux based systems)
- Clone github repo:
git clone https://github.com/cantonioupao/3d-virtual-chatbot/
If you find our work useful to your research, please consider citing:
For the Colab notebooks of external models, we aknolwedge that the codebase and scripts used within these notebooks do not belong to us. Here are some great resources we benefit from:
- fs_vid2vid - Few-shot vid2vid
- VOCA - Voice Operated Character Animation
- DECA - Detailed Expression Capture and Animation (SIGGRAPH2021)
- StyleGAN3 - Alias-Free Generative Adversarial Networks (StyleGAN3) (NeurIPS 2021)
- unsup3D - Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild
- NHA - Neural Head Avatars - Neural Head Avatars from Monocular RGB Videos