PyTorch implementation for paper ShaDDR: Interactive Example-Based Geometry and Texture Generation via 3D Shape Detailization and Differentiable Rendering, Qimin Chen, Zhiqin Chen, Hang Zhou, Hao Zhang.
Paper | Project page | Video
If you find our work useful in your research, please consider citing (to be updated):
@misc{chen2023shaddr,
title={ShaDDR: Real-Time Example-Based Geometry and Texture Generation via 3D Shape Detailization and Differentiable Rendering},
author={Qimin Chen and Zhiqin Chen and Hang Zhou and Hao Zhang},
year={2023},
eprint={2306.04889},
archivePrefix={arXiv}
}
Requirements:
- Python 3.7 with numpy, pillow, h5py, scipy, sklearn and Cython
- PyTorch 1.9 (other versions may also work)
- PyMCubes (for marching cubes)
- OpenCV-Python (for reading and writing images)
Build Cython module:
python setup.py build_ext --inplace
For data preparation, please see data_preparation.
We provide the ready-to-use datasets here.
We also provide the pre-trained network weights.
Make sure to train the geometry detailization first, then train texture generation.
python main.py --data_style style_color_car_16 --data_content content_car_train --data_dir ./data/02958343/ --alpha 0.2 --beta 10.0 --input_size 64 --output_size 512 --train --train_geo --gpu 0 --epoch 20
python main.py --data_style style_color_car_16 --data_content content_car_train --data_dir ./data/02958343/ --alpha 0.2 --beta 10.0 --input_size 64 --output_size 512 --train --train_tex --gpu 0 --epoch 20
python main.py --data_style style_color_plane_16 --data_content content_plane_train --data_dir ./data/02691156/ --alpha 0.1 --beta 10.0 --input_size 64 --output_size 512 --train --train_geo --gpu 0 --epoch 20
python main.py --data_style style_color_plane_16 --data_content content_plane_train --data_dir ./data/02691156/ --alpha 0.2 --beta 10.0 --input_size 64 --output_size 512 --train --train_tex --gpu 0 --epoch 20
python main.py --data_style style_color_chair_16 --data_content content_chair_train --data_dir ./data/03001627/ --alpha 0.3 --beta 10.0 --input_size 32 --output_size 256 --train --train_geo --gpu 0 --epoch 20
python main.py --data_style style_color_chair_16 --data_content content_chair_train --data_dir ./data/03001627/ --alpha 0.2 --beta 10.0 --input_size 32 --output_size 256 --train --train_tex --gpu 0 --epoch 20
python main.py --data_style style_color_building_8 --data_content content_building_train --data_dir ./data/00000000/ --alpha 0.5 --beta 10.0 --input_size 32 --output_size 256 --train --train_geo --asymmetry --gpu 0 --epoch 20
python main.py --data_style style_color_building_8 --data_content content_building_train --data_dir ./data/00000000/ --alpha 0.2 --beta 10.0 --input_size 32 --output_size 256 --train --train_tex --asymmetry --gpu 0 --epoch 20
These are examples for testing a model trained with 16 detailed cars. For other categories, please change the commands accordingly.
To output a few detailization results:
python main.py --data_style style_color_car_16 --data_content content_car_test --data_dir ./data/02958343/ --input_size 64 --output_size 512 --test --test_tex --gpu 0
Please add --asymmetry
if testing the building category since we train the building category without using the symmetry assumption.
The output mesh can be found in folder samples or you can specify --sample_dir
.
(Borrowed from DECOR-GAN) To test Strict-IOU, Loose-IOU, LP-IOU, Div-IOU, LP-F-score, Div-F-score:
python main.py --data_style style_color_chair_16 --data_content content_chair_test --data_dir ./data/03001627/ --input_size 32 --output_size 256 --prepvoxstyle --gpu 0
python main.py --data_style style_color_chair_16 --data_content content_chair_test --data_dir ./data/03001627/ --input_size 32 --output_size 256 --prepvox --gpu 0
python main.py --data_style style_color_chair_16 --data_content content_chair_test --data_dir ./data/03001627/ --input_size 32 --output_size 256 --evalvox --gpu 0
The first command prepares the patches in 16 detailed training shapes, thus --data_style is style_color_chair_16. Specifically, it removes duplicated patches in each detailed training shape and only keeps unique patches for faster computation in the following testing procedure. The unique patches are written to the folder unique_patches. Note that if you are testing multiple models, you do not have to run the first command every time -- just copy the folder unique_patches or make a symbolic link.
The second command runs the model and outputs the detailization results, in folder output_for_eval.
The third command evaluates the outputs. The results are written to folder eval_output ( result_IOU_mean.txt, result_LP_Div_Fscore_mean.txt, result_LP_Div_IOU_mean.txt ).
(Borrowed from DECOR-GAN) To test Cls-score:
python main.py --data_style style_color_chair_16 --data_content content_chair_all --data_dir ./data/03001627/ --input_size 32 --output_size 256 --prepimgreal --gpu 0
python main.py --data_style style_color_chair_16 --data_content content_chair_test --data_dir ./data/03001627/ --input_size 32 --output_size 256 --prepimg --gpu 0
python main.py --data_style style_color_chair_16 --data_content content_chair_all --data_dir ./data/03001627/ --input_size 32 --output_size 256 --evalimg --gpu 0
The first command prepares rendered views of all content shapes, thus --data_content is content_chair_all. The rendered views are written to the folder render_real_for_eval. Note that if you are testing multiple models, you do not have to run the first command every time -- just copy the folder render_real_for_eval or make a symbolic link.
The second command runs the model and outputs rendered views of the detailization results, in folder render_fake_for_eval.
The third command evaluates the outputs. The results are written to folder eval_output ( result_Cls_score.txt ).
- Build Cython module:
cd gui
python setup.py build_ext --inplace
- Make sure you put the checkpoint.pth in the
checkpoint
folder, checkpoint can be found here - Change the
cpk_path
in thegui_demo.py
- Run the GUI
python gui_demo.py --category 00000000
- Some basic modeling operations of GUI
add voxel - ctrl + left click
delete voxel - shift + left click
rotate - left click + drag
zoom in/out - scroll wheel
GUI currently only supports editing voxel from scratch, more input formats will be supported in the future.
https://github.com/qiminchen/ShaDDR/blob/main/gui/shaddr_demo.mp4