GSOC 2024: 3D Gaussian Splat for point cloud generation with MPM simulations

[kks32]

3D Gaussian Splat for Point Cloud Generation in Physics-Driven MPM Simulations

Abstract

This project aims to bridge the gap between real-world phenomena and computational physics simulations by converting video data of dynamic events, such as granular column collapses or dam breaks, into point clouds in real time using Gaussian splatting techniques. The generated point clouds will then be utilized in Material Point Method (MPM) simulations to model and understand these complex scenarios' underlying physics accurately. This approach will integrate with the CB-Geo MPM project, focusing on rendering natural hazards like landslides with an in-situ visualization interface. The project holds high priority due to its potential to enhance predictive modeling and digital twin reconstructions of natural disasters, contributing significantly to computational geotechnics and disaster management.

Intensity	Priority	Involves	Mentors
Moderate	High	Integrating Gaussian splatting with existing rendering systems in CB-Geo MPM to generate 3D point clouds of natural hazards such as landslides.	Krishna Kumar and Justin Bonus

Benefits of working on this project

Students engaging in this project will enhance their skills in:

• Neural rendering frameworks and text-to-mesh generation
• Application of machine learning techniques in physics-based models.
• Working with advanced rendering and simulation tools in natural hazard engineering.

Motivation

Current techniques for modeling natural hazards and other dynamic events often rely on static datasets that do not capture the full scope of real-world variability. This project aims to create more dynamic and accurate digital twins of natural hazards by leveraging real-time video data and converting it into point clouds for MPM simulations. This method will enable the prediction and analysis of complex phenomena with unprecedented detail and fidelity.

Technical Details

• Gaussian Splatting: A technique used for converting video data into point clouds by projecting video pixels into 3D space, using a Gaussian function to manage the distribution of points.
• Material Point Method (MPM) Simulations: A physics-based computational method used for capturing the behavior of materials under various conditions, ideal for simulating natural disasters and hazards.
• The project will integrate these technologies to convert real-world video footage into detailed simulations, aiming to minimize the loss between observed and simulated data to identify accurate physical models.

Benefits to Project/Community

This project will significantly contribute to the fields of robotics, natural hazard prediction, and digital twin reconstruction by:

• Offering a novel method for real-time data conversion and simulation.
• Enhancing the accuracy and applicability of predictive models for natural disasters.
• Providing a robust framework for integrating video data with physical simulations.

Helpful Experience

Candidates interested in this project should ideally:

• Have a working knowledge of C++ and Python.
• Be experienced or keen on rendering or visualization tools like OpenUSD, Houdini, or Blender.

First Steps

Prospective participants should:

• Review the foundational paper describing the Gaussian splatting process and its application in MPM simulations.
• Acquaint themselves with existing rendering engines like Blender, the OpenUSD framework, and CB-Geo MPM's current simulation and visualization infrastructure.
• Gain familiarity with conducting petascale simulations on high-performance computing resources like the Texas Advanced Computing Center (TACC) supercomputer.