Real-Time Free-viewport Holographic Patient APP with HumanNeRF and Gaussian Splatting techniques
Supplementary video:https://youtu.be/37sN7_GQOdo
- upload video processing codes to estiamte 3D pose, masks, and camera parameters
- release testing code
- release trained weights
- release training code
- release server code
- release Unity scripts
- release executable file
NVIDIA GPUs are required for this project.
Linux server: The implementation of the server code is tested on:
Distributor ID: Ubuntu
Description: Ubuntu 18.04.6 LTS
Release: 18.04
Codename: bionic
Unity client is tested on windows 11
- simple_romp https://github.com/Arthur151/ROMP, we modified the package and make it able to produce SMPL masks
- simple_knn https://github.com/camenduru/simple-knn
- diff-gaussian-rasterazation https://github.com/graphdeco-inria/diff-gaussian-rasterization
Register and download SMPL models here. Put the downloaded models in the folder smpl_models. Only the neutral one is needed.
humanModel
-- assets
----SMPL_NEUTRAL.pkl
-- smpl
---- smpl_numpy.py
-- smpl-meta
---- faces.npy
---- J_regressor_body25.npy
---- parents.npy
---- SMPL_NEURAL.pkl
---- smpl_uv.obj
---- weights.npy
-- smplx
conda create -n RTFVHP python=3.8
pip install Cython==3.0.10
pip install numpy==1.24.1
pip install opencv-python==4.9.0.80
pip install tqdm
pip install chumpy
# comment out the line in chumpy __init__.py :
# from numpy import bool, int, float, complex, object, unicode, str, nan, inf
pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2
cd RTFVHP/submodules/ROMP/simple_romp python setup.py install