Physics3D

Official implementation of Physics3D: Learning Physical Properties of 3D Gaussians via Video Diffusion

Fangfu Liu, Hanyang Wang, Shunyu Yao, Shengjun Zhang, Jie Zhou, Yueqi Duan

Paper | Project page | Data

Physics3D is a unified simulation-rendering pipeline based on 3D Gaussians, which learn physics dynamics from video diffusion model.

More features

The repo is still being under construction, thanks for your patience.

Training code release.
Synthetic data release.
Detailed tutorial.
Detailed local demo.

Preparation for training

Linux System Setup.

conda create -n Physics3D python=3.9
conda activate Physics3D

pip install -r requirements.txt
git clone https://github.com/graphdeco-inria/gaussian-splatting
pip install -e gaussian-splatting/submodules/diff-gaussian-rasterization/
pip install -e gaussian-splatting/submodules/simple-knn/

Quick Start.

Download the Gaussian models from OneDrive. You can also load your own 3D Gaussian pre-trained models to this pipeline following gaussian-splatting. For the setting details of physical configs, you can refer to PhysGaussian.
```
Physics3D
    ├──model
        ├── ball/
    ├──config
        ├── ball_config.json
```

We support using text-to-video (ModelScope) diffusion models to guide the optimization of physical parameters. You can use the following command:

python simulation.py --model_path ./model/ball/ --prompt "a basketball falling down" --output_path ./output --physics_config ./config/ball_config.json

Tips to get better results

Parameter initialization that aligns with physical facts can significantly accelerate the convergence of Physics3D and improve training effectiveness.
For some high-frequency elastic objects, simulation effectiveness can be enhanced by increasing particle density.

Acknowledgement

We have intensively borrowed code from the following repositories. Many thanks to the authors for sharing their code.

DreamPhysics
threestudio and its extension for Animate124.
warp-mpm
PhysGaussian

We have also used open-source datasets from the following repositories.

PhysDreamer
BlenderKit for free models and BlenderNeRF for synthetic NeRF datasets within Blender

Citation

If you found Physics3D helpful, please cite our report:

@misc{liu2024physics3d,
      title={Physics3D: Learning Physical Properties of 3D Gaussians via Video Diffusion}, 
      author={Fangfu Liu and Hanyang Wang and Shunyu Yao and Shengjun Zhang and Jie Zhou and Yueqi Duan},
      year={2024},
      eprint={2406.04338},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Contact

If you have any question about this project, please feel free to contact liuff23@mails.tsinghua.edu.cn or hanyang-21@mails.tsinghua.edu.cn.

alexandor91/Physics3D