Motion2VecSets: A Python repository from VVeiCao

🐎Motion2Vecsets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and Tracking

Wei Cao^1* · Chang Luo^1* · Biao Zhang²
Matthias Nießner¹ · Jiapeng Tang¹

¹Technical University of Munich · ²King Abdullah University of Science and Technology
_{(* indicates equal contribution)}

CVPR 2024

[Arxiv] [Paper] [Project Page] [Video]

We present Motion2VecSets, a 4D diffusion model for dynamic surface reconstruction from sparse, noisy, or partial point cloud sequences. Compared to the existing state-of-the-art method CaDeX, our method can reconstruct more plausible non-rigid object surfaces with complicated structures and achieve more robust motion tracking.

Given a sequence of sparse and noisy point clouds as inputs, labeled as P^t for t from 1 to T, Motion2VecSets outputs a continuous mesh sequence, labeled as M^t for t from 1 to T. The initial input frame, P¹ (top left), is used as a condition in the Shape Vector Set Diffusion, which yields denoised shape codes, S, for reconstructing the geometry of the reference frame, M¹ (top right). Concurrently, the subsequent input frames, P^t for t from 2 to T (bottom left), are utilized in the Synchronized Deformation Vector Sets Diffusion to produce denoised deformation codes, D^t for t from 2 to T, where each latent set, D^t, encodes the deformation from the reference frame, M¹, to each subsequent frame, M^t.

Install

Install the environment following scripts/init_environment.sh, to install with cuda 11.0, use the command bash scripts/init_environment.sh

Data preparation and Pretrained models

You can find the data preparation scripts in /home/liang/m2v/dataset/dataset_generate or you can directly download the preprocessed dataset.

For pretrained models, you can directly put them in the ckpts directory, and there are two subfolders DFAUST and DT4D for two datasets respectively, you can get them from the Google Drive.

Training and evaluation

You can run the training with the following command, here we use the DFAUST dataset as an example. For DT4D dataset you may just change the path of config file.

# Shape
python core/run.py --config_path ./configs/DFAUST/train/dfaust_shape_ae.yaml # Shape AE
python core/run.py --config_path ./configs/DFAUST/train/dfaust_shape_diff_sparse.yaml # Diffusion Sparse Input
python core/run.py --config_path ./configs/DFAUST/train/dfaust_shape_diff_partial.yaml # Diffusion Partial Input

# Deformation
python core/run.py --config_path ./configs/DFAUST/train/dfaust_deform_ae.yaml # Deformation AE
python core/run.py --config_path ./configs/DFAUST/train/dfaust_deform_diff_sparse.yaml # Diffusion Sparse Input
python core/run.py --config_path ./configs/DFAUST/train/dfaust_deform_diff_partial.yaml # Diffusion Partial Input

For evaluation, you can run the following command:

python core/eval.py --config_path ./configs/DFAUST/eval/dfaust_eval_sparse.yaml # Test Sparse Unssen Sequence
python core/eval.py --config_path ./configs/DFAUST/eval/dfaust_eval_sparse.yaml --test_ui # Test Sparse Unssen Individual

python core/eval.py --config_path ./configs/DFAUST/eval/dfaust_eval_partial.yaml # Test Partial Unssen Sequence
python core/eval.py --config_path ./configs/DFAUST/eval/dfaust_eval_partial.yaml --test_ui # Test Partial Unssen Individual

Demo

You can run demo/infer.py to get the predicted mesh sequence for the inputs located in demo/inputs, before running the demo, you need to download the pretrained models and put them in the ckpts directory.

Citation

@inproceedings{Wei2024M2V,
    title={Motion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and Tracking},
    author={Wei Cao and Chang Luo and Biao Zhang and Matthias Nießner and Jiapeng Tang},
    booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
    url={https://vveicao.github.io/projects/Motion2VecSets/},
    year={2024}
}

Reference

This project is based on several wonderful projects:

3DShape2Vecset: https://github.com/1zb/3DShape2VecSet
CaDeX: https://github.com/JiahuiLei/CaDeX
LPDC: https://github.com/Gorilla-Lab-SCUT/LPDC-Net
Occupancy Flow: https://github.com/autonomousvision/occupancy_flow
DeformingThings4D: https://github.com/rabbityl/DeformingThings4D

VVeiCao/Motion2VecSets