This is a Nerfstudio implementation of the Fuzzy Metaball rendering methods. The enable optimization with purely color-based losses, this includes a far-field NeRF to model the background. The Gaussians then organize into modeling just the foreground object.
The Fuzzy Metaballs project focused on shape optimization, with no priors, with very few 3D Gaussians (for robustness, simplicitly, speed, etc.), so this implementation defaults to using 40 Gaussians. This produces novel view synthesis that is worse NeRF methods (e.g. about 18 to 20 PSNR), but reasonable, simplified geometric reconstructions. Adding smarter colors to each Gaussian (e.g. spherical harmonics, MLPs, or textures) would certainly improve PSNR, but isn't done for this example.
Example reconstruction of Nerfstudio plane and sculpture sequences with 40 Gaussians. This using posed color images (no masks, no flow) and random initialization from a small sphere, with a few thousand iterations of optimization.
plane_40.mp4
sculpture_40.mp4
You can see how the forty Gaussians distribute themselves to model the foreground. Of note, some Gaussians are used to model the shadows and the specularities and other color-specific features. The default learning rates are reasonable for real sequences, but may be too high for synthetic sequences with no real backgrounds.
First, install nerfstudio. Second, Clone or fork this repository and run the commands:
pip install -e .
ns-install-cli
This repository creates a new Nerfstudio method named "fmb". To train with it, run the command:
ns-train fmb --data [PATH]
- Good: Color-based optimization works! This wasn't tested in either prior paper.
- Good: Many background models work. NeRF interpolates well, but even a texture atlas works well on i.i.d. evaluation.
- Bad: It is slow. The current torch implementation seems imperfect. On the same machine, this might get 150-300K rays per second (with simple background), but the JAX codebase gets around 2M train ray/s (even with extra losses like flow!).