/avatar_ernerf

Just a suturing monster project.

Primary LanguagePython

Avatar ERNeRF

ER-NeRF : Paper | github

Language: [English] | [简体中文]

Overview

  • This is just a suturing monster project, a whole process scheme of avatar video generation based on ER-NeRF algorithm to provide additional pre-processing and post-processing. The author is not the author of ER-NeRF algorithm, and does not have the ability to answer any problem with poor processing effect.

Installation

  • Ubuntu18.04; CUDA11.3; CUDNN>=8.2.4, <8.7.0; gcc/g++-9;

  • third party:

    sudo apt-get install libasound2-dev portaudio19-dev # dependency for pyaudio
sudo apt-get install ffmpeg # or build from source
# build openface from source `https://github.com/TadasBaltrusaitis/OpenFace.git`

  • python env:

    conda create -n ernerf python=3.10 -y
conda activate ernerf

# install pytorch
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113

# install pytorch3d
pip install "git+https://github.com/facebookresearch/pytorch3d.git"

# install others
pip install -r requirements.txt

sh install_ext.sh

  • pretrained weights:

    Download the pretrained weights from: Google Drive

    Download 3DMM model for head pose estimation:

    wget https://github.com/YudongGuo/AD-NeRF/blob/master/data_util/face_tracking/3DMM/exp_info.npy?raw=true -O pretrained_weights/3DMM/exp_info.npy
wget https://github.com/YudongGuo/AD-NeRF/blob/master/data_util/face_tracking/3DMM/keys_info.npy?raw=true -O pretrained_weights/3DMM/keys_info.npy
wget https://github.com/YudongGuo/AD-NeRF/blob/master/data_util/face_tracking/3DMM/sub_mesh.obj?raw=true -O pretrained_weights/3DMM/sub_mesh.obj
wget https://github.com/YudongGuo/AD-NeRF/blob/master/data_util/face_tracking/3DMM/topology_info.npy?raw=true -O pretrained_weights/3DMM/topology_info.npy

    Download 3DMM model from Basel Face Model 2009:

    # 1. copy 01_MorphableModel.mat to `pretrained_weights/3DMM`
# 2.
cd modules/face_tracking
python convert_BFM.py

Usage

  • The input video data should be a video of about 5-10 minutes, there is only one person in the video, and it needs to ensure time continuity (one shot to the end).

    cd <PROJECT>
export PYTHONPATH=./

python -u create_train_task.py -i <input_video> --model_uid <model_name>

python -u create_infer_task.py -i <input_audio> -c <model_name or model_config_file>

Acknowledgement