/UnityGaussianSplatting

Toy Gaussian Splatting visualization in Unity

Primary LanguageC#MIT LicenseMIT

Toy Gaussian Splatting playground in Unity

SIGGRAPH 2023 had a paper "3D Gaussian Splatting for Real-Time Radiance Field Rendering" by Kerbl, Kopanas, Leimkühler, Drettakis that looks pretty cool! Check out their website, source code repository, data sets and so on.

I've decided to try to implement the realtime visualization part (i.e. the one that takes already-produced gaussian splat "model" file) in Unity.

Screenshot

The original paper code has a purely CUDA-based realtime renderer; other people have done their own implementations (e.g. WebGPU at cvlab-epfl, Taichi at wanmeihuali, etc.).

Code in here so far is randomly cribbled together from reading the paper (as well as earlier literature on EWA splatting), looking at the official CUDA implementation, and so on. Current state:

  • The code does not use the "tile-based splat rasterizer" bit from the paper; it just draws each gaussian splat as a screenspace aligned rectangle that covers the extents of it.
  • Splat color accumulation is done by rendering front-to-back, with a blending mode that results in the same accumulated color as their tile-based renderer.
  • Splat sorting is done with a AMD FidelityFX derived radix sort.

Usage

⚠️ Note: this is all a toy, it can be not robust, not handle errors, not composite well with the rest of rendering, not be fast, etc. Also, do not file bugs or issues just yet; I will most likely just ignore them and do whatever I please. I told you so! ⚠️

First download or clone this repository and open as a Unity (2022.3, other versions might also work) project. Note that the project requires DX12 or Vulkan on Windows, i.e. DX11 will not work.

Next up, create some GaussianSplat assets: open Tools -> Gaussian Splats -> Create GaussianSplatAsset menu within Unity. In the dialog, point Input PLY File to your Gaussian Splat file (note that it has to be a gaussian splat PLY file, not some other PLY file. E.g. in the official paper models, the correct files are under point_cloud/iteration_*/point_cloud.ply). Optionally there can be cameras.json next to it or somewhere in parent folders.

Pick desired compression options and output folder, and press "Create Asset" button. The compression even at "very low" quality setting is decently usable, e.g. this capture at Very Low preset is under 8MB of total size (click to see the video):
Watch the video

If everything was fine, there should be a GaussianSplat asset that has several data files next to it.

Since the gaussian splat models are quite large, I have not included any in this Github repo. The original paper github page has a a link to 14GB zip of their models.

In the game object that has a GaussianSplatRenderer script, point the Asset field to one of your created assets. There are various controls on the script to debug/visualize the data, as well as a slider to move game camera into one of asset's camera locations.

The rendering takes game object transformation matrix into account; the official gaussian splat models seem to be all rotated by about -160 degrees around X axis, and mirrored around Z axis, so in the sample scene the object has such a transform set up.

In the built-in render pipeline, the gaussian splatting should work with no extra steps. If you are using URP, add GaussianSplatURPFeature to the URP renderer settings. If you are using HDRP, add CustomPass volume object and a GaussianSplatHDRPPass entry to it. Maybe also set injection point to "after postprocess" to stop auto-exposure from going wild.

When a GaussianSplatRenderer object is selected, there's an additional tool that shows up in the scene view to edit the splats. You can rectangle-drag to select them (shift+drag adds to selection). Usual Select All, Invert Selection etc. shortcuts work too. Delete/Backspace deletes the selected splats. In the inspector there's a button then to export the "edited" object back into a Gaussian Splat PLY file. This is best done using Very High import option for the original splat PLY file.

That's it!

Wishlist that I may or might not do at some point:

  • Investigate hashed alpha testing instead of blending (removes need for sorting splats)
  • Make low quality levels work on mobile (look into ASTC texture compression?)
  • Make a C/WebAssembly library to do PLY quantization/compression just like in Unity
  • Make a WebGL/WebGPU example that uses the smaller data files
  • Make rendering faster (actual tiled compute shader rasterizer)

Write-ups

My own blog posts about all this:

Performance numbers:

"bicycle" scene from the paper, with 6.1M splats and first camera in there, rendering at 1200x797 resolution, at "Medium" asset quality level (283MB asset file):

  • Windows (NVIDIA RTX 3080 Ti):
    • Official SBIR viewer: 7.4ms (135FPS). 4.8GB VRAM usage.
    • Unity, DX12 or Vulkan: 12.6ms (79FPS) - 9.4ms rendering, 2.4ms sorting, 0.7ms splat view calc. 1.2GB VRAM usage.
  • Mac (Apple M1 Max):
    • Unity, Metal: 31.8ms (31FPS).

Besides the gaussian splat asset that is loaded into GPU memory, currently this also needs about 48 bytes of GPU memory per splat (for sorting, caching view dependent data etc.).

License and External Code Used

The code I wrote for this is under MIT license. The project also uses several 3rd party libraries:

However, the license of the original paper is not quite clear to me. In this project, I have not used their code, but I have read their paper and have seen their code. I've no idea if that somehow makes this project be under their license too? Someone should figure this out.