Immersive Neural Graphics Primitives

In this project, we present immersive NGP, the first open-source VR NERF Unity package that brings high resolution, low-latency, 6-DOF NERF rendering to VR. This work is based on Nvidia's ground breaking instant-ngp technique. Current version uses this commit of instant-ngp.

Features

Stereoscopic, 6-DOF, real-time, low-latency NERF VR rendering in Unity.
DLSS Support for rendering at higher framerate.
6-DOF continuous locomotion in VR.
Offline volume image slices rendering via Unity Volume Rendering Toolkit.
Integration with MRTK 2.8 for building mixed reality applications with NERF.

Dependencies

Unity 2019.4.29 ( Use the legacy XR manager for compatibility with OpenVR)
instant-ngp
Unity OpenVR desktop plugin && SteamVR
Microsoft Mixed Reality Toolkit MRTK 2.8 (already included in the Unity project)
OpenGL Graphics API
Current version of the repository was tested on Windows 10, Windows 11, using a Oculus Quest 2.

Installation

Clone this repository: git clone --recursive https://github.com/uhhhci/immersive-ngp
Make sure you have all the dependencies for instant-ngp installed before proceed.

Update dependencies for submodules

git submodule sync --recursive
git submodule update --init --recursive

Build the instant-ngp project, similar to the build process for the original instant-ngp project.
```
cmake . -B build
cmake --build build --config RelWithDebInfo -j
```
After succesful build, copy the following plugin files from \instant-ngp\build\ folder to the \stereo-nerf-unity\Assets\Plugins\x86_64 folder.
Now instant-ngp can be loaded as native plugins via Unity.

Usage for Immersive NERF Rendering

For Oculus Quest 2, Lunch Oculus Rift, and connect the headset to the PC via Link Cable, or Air Link.
Launch SteamVR, make sure that SteamVR detects your headset and controllers.
For a quick demo train a model using the fox scene via:
```
build\testbed.exe --scene ..\data\nerf\fox
```
and safe a snapshot of the instant-ngp model through Instant-ngp > Snapshot > Save
Open the stereo-nerf-unity Unity project with Unity 2019.4.29.
For a quick VR test of your own NERF scene, go to the Assets\NERF_NativeRendering\Scenes\XRTest scene.
Copy the path to your nerf model, images folder, and transform.json file to the Stereo Nerf Renderer in the Nerf path parameters, as ilustrated below.

$.\images\stereo-nerf-gameobj.PNG$

(Note: please generate the nerf model using this instant-ngp commit and above, or just use the instant-ngp instance included in this repo).
Adjust DLSS settings, and image resolution as you like.
Now you can run the scene in Editor :)
Use the joystick of the VR controllers for locomotion.

Common Questions & Troubleshoot

How to reach good framerate and lower latency

Beside having a good GPU, it is highly recommended to turn on DLSS support in Unity, also when building the native plugin.

The instant-ngp commit we use also allows saving aabb cropping in the pre-trained model snapshot. If you adjust the aabb cropping when training the model, it will saved and be loaded in Unity as well. Reducing aabb cropping could reduce the render volume, thus save some computational power.
Locomotion doesn't work.

Make sure that SteamVR detects both of your controllers before starting the scenes in the Editors.
How to change the FoV the image frame.

You can customize the FoV or aspect ratio of the stereo NERF image plane by modifying the x and y scale of the NERFLeft Image and the NERFRight Image game object. (Make sure to adjust both at the same time).

$.\images\stereo-nerf-gameobj.PNG$
Is VR headset required to run the demo?

Yes, the main focus of this repository target immersive applications with a VR headset + steamVR. Please feel free to fork this repo and build immersive desktop applications.

Updates

2023/01/05: Major refractoring and fixed the issue that editor needs to be restarted on second run.

Roadmap

~~Fix Editor restart issue~~
Foveated NERF : under consturction, coming soon ...
Automatic Fov adjutsment: under consturction, coming soon ...
Various interaction techniques (egocentric manipulation, etc) : under consturction, coming soon ...
Time-warp algorithm for latency compensation: welcome contribution ...
Dynamics Resolution: welcome contribution
Support for OpenXR: welcome contribution
Support for higher Unity Version: welcome contribution
Real-time SLAM capture for dynamic grow dataset: welcome contribution
.... feature suggestions welcome!

Contributions

We welcome community contributions to this repository.

Thanks

Many thanks to the authors of these open-source repositories:

Citations

@misc{immersive-ngp,
      doi = {10.48550/ARXIV.2211.13494},
      url = {https://arxiv.org/abs/2211.13494},
      author = {*Li, Ke and *Rolff, Tim and Schmidt, Susanne and Bacher, Reinhard and Frintrop, Simone and Leemans, Wim and Steinicke, Frank},
      title = {Immersive Neural Graphics Primitives},
      publisher = {arXiv},
      year = {2022}}

*These authors contributed equally to the work.

Link to Arxiv paper

Contact: ke.li1@desy.de, tim.rolff@uni-hamburg.de

Acknowledgment

This work was supported by DASHH (Data Science in Hamburg - HELMHOLTZ Graduate School for the Structure of Matter) with the Grant-No. HIDSS-0002, and the German Federal Ministry of Education and Research (BMBF).

License

Please check here to view a copy of Nvidia's license for instant-ngp and for this repository.

NessaVR/immersive-ngp