Immersive Neural Graphics Primitives

In this project, we present immersive NGP, the first open-source VR NERF Unity package that brings high resolution, low-latency, 6-DOF NERF rendering to VR. This work is based on Nvidia's ground breaking instant-ngp technique. Current version uses this commit of instant-ngp.

Features

Stereoscopic, 6-DOF, real-time, low-latency NERF VR rendering in Unity.
DLSS Support for rendering at higher framerate.
6-DOF continuous locomotion in VR.
Offline volume image slices rendering via Unity Volume Rendering Toolkit.
Integration with MRTK 2.8 for building mixed reality applications with NERF.
- Example: Merging a NERF volume image slices with a CAD model

Dependencies

Unity 2019.4.29 ( Use the legacy XR manager for compatibility with OpenVR)
instant-ngp
Unity OpenVR desktop plugin && SteamVR
Microsoft Mixed Reality Toolkit MRTK 2.8
OpenGL Graphics API
Current version of the repository was tested on Windows 10, and Oculus Quest 2.

Installation

Clone this repository: git clone --recursive https://github.com/uhhhci/immersive-ngp
Make sure you have all the dependencies for instant-ngp installed before proceed.

Update dependencies for submodules

git submodule sync --recursive
git submodule update --init --recursive

Build the instant-ngp project, similar to the build process for the original instant-ngp project.
```
cmake . -B build
cmake --build build --config RelWithDebInfo -j
```
After succesful build, copy the following plugin files from \instant-ngp\build\ folder to the \stereo-nerf-unity\Assets\Plugins\x86_64 folder.
Now instant-ngp can be loaded as native plugins via Unity.

Usage for Immersive NERF Rendering

Open the stereo-nerf-unity Unity project with Unity 2019.4.29.
For a quick VR test of your own NERF scene, go to the Assets\NERF_NativeRendering\Scenes\XRTest scene.
Copy the path to your nerf model, images folder, and transform.json file to the Stereo Nerf Renderer in the Nerf path parameters, as ilustrated below.

$.\images\stereo-nerf-gameobj.PNG$
Adjust DLSS settings, and image resolution as you like.
Now you can run the scene in Editor :)
Disclaimer: There is currently a small issue with running the scene in Unity Editor with native plugin clean up, you might need to restart the editor when running a new scene.

Roadmap

Fix Editor restart issue
Time-warp algorithm for latency compensation
Dynamics Resolution
Foveated NERF
Support for OpenXR
Support for higher Unity Version
Real-time SLAM capture for dynamic grow dataset

Thanks

Many thanks to the authors of these open-source repositories:

Authors

*Ke Li^{1, 2}, * Tim Rolff^1,3, Susanne Schmidt ¹, Reinhard Bacher ² , Simone Frintrop ³ , Wim Leemans ³ , Frank Steinicke ¹

*These authors contributed equally to the work.

¹ Human-Computer Interaction Group, Department of Informatics, Universität Hamburg

² Deutsches Elektronen-Synchrotron DESY, Germany

³ Computer Vision Group, Department of Informatics, Universität Hamburg

Contact: ke.li1@desy.de, tim.rolff@uni-hamburg.de

Citations

@misc{Immersive-NGP,
  author = {Tim Rolff and Ke Li and Susanne Schmidt and Reinhard Bacher and   Simone Frintrop and Wim Leemans and Frank Steinicke},
  title = {Immersive Neural Graphics Primitives},
  year = {2022},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/uhhhci/Immersive-Neural-Graphics-Primitives}},
}

Acknowledgment

This work was supported by DASHH (Data Science in Hamburg - HELMHOLTZ Graduate School for the Structure of Matter) with the Grant-No. HIDSS-0002, and the German Federal Ministry of Education and Research (BMBF).

License

Please check here to view a copy of Nvidia's license for instant-ngp and for this repository.

sonicviz/unity-immersive-ngp