Junsheng Zhou* · Weiqi Zhang* · Baorui Ma · Kanle Shi · Yu-Shen Liu · Zhizhong Han
(* Equal Contribution)
We will release the code of the paper UDiFF: Generating Conditional Unsigned Distance Fields with Optimal Wavelet Diffusion in this repository.
In this work, we present UDiFF, a 3D diffusion model for unsigned distance fields (UDFs) which is capable to generate textured 3D shapes with open surfaces from text conditions or unconditionally. Our key idea is to generate UDFs in spatial-frequency domain with an optimal wavelet transformation, which produces a compact representation space for UDF generation. Specifically, instead of selecting an appropriate wavelet transformation which requires expensive manual efforts and still leads to large information loss, we propose a data-driven approach to learn the optimal wavelet transformation for UDFs.
Overview of UDiFF. (a) We propose a data-driven approach to attain the optimal wavelet transformation for UDF generation. We optimize wavelet filter parameters through the decomposition and inversion by minimizing errors in UDF self-reconstruction. (b) We fix the learned decomposition wavelet parameters and leverage it to prepare the data as a compact representation of UDFs including pairs of coarse and fine coefficient volumes. (c) is the architecture of the generator in diffusion models, where text conditions are introduced with cross-attentions. (d) The diffusion process of UDiFF. We train the generator to produce coarse coefficient volumes from random noises guided by input texts and train the fine predictor to predict fine coefficient volumes from the coarse ones.
Outfit Designs with UDiFF Garment Generations
We recommend creating an anaconda environment using our provided environment.yml:
conda env create -f environment.yml
conda activate udiff
pip install git+https://github.com/fbcotter/pytorch_wavelets
We use point clouds in the Deepfashion dataset.
Please modify the root path in sample_udf.py and run it.
After sampling the UDF from the point cloud, please run learn_wavelet/train.py to train the wavelet parameters.
Finally, you can run convert_wavelet.py using the checkpoint trained in the previous step. It should be noted that setting the resolution_index to 3 will generate the coarse wavelet coefficients, whereas setting it to 2 will generate the detail wavelet coefficients.
At this point you can start training unconditional, if you need to generate text or image conditions, please refer to openshape.
The process of training is primarily governed by the configuration script (config.py or config_highs.py).
It is essential to define data_files direction to the two npy files that have been created.
-
Train diffusion model
python trainer/trainer.py --resume_path ./configs/config.py -
Train detail predictor
python trainer/trainer.py --resume_path ./configs/config_highs.py
To execute inference, please modify models/network_gen.py by updating the diffusion_folder and high_level_folder. Additionally, you also need to adjust the epoch numbers.
And you can run the following command:
python models/network_gen.py
If you want to generate high quality textures for generated meshes, please refer to Text2Tex.
We provide the pretrained models: learned wavelet, diffusion model and detail predictor of the unconditional model. Please download the pretrained models from Google Drive.
If you find our code or paper useful, please consider citing
@inproceedings{udiff,
title={UDiFF: Generating Conditional Unsigned Distance Fields with Optimal Wavelet Diffusion},
author={Zhou, Junsheng and Zhang, Weiqi and Ma, Baorui and Shi, Kanle and Liu, Yu-Shen and Han, Zhizhong},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year={2024}
}