/Make_it_Real

[NeurIPS 2024] Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials

Primary LanguagePythonMIT LicenseMIT

Make-it-Real

Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials
Ye Fang*, Zeyi Sun*, Tong Wu, Jiaqi Wang, Ziwei Liu, Gordon Wetzstein, Dahua Lin

*Equal Contribution

Demo

πŸ“œ News

πŸš€ [2024/6/8] We release our inference pipeline of Make-it-Real, including material matching and generation of albedo-only 3D objects.

πŸš€ [2024/6/8] Material library annotations generated by GPT-4V and data engine are released!

πŸš€ [2024/4/26] The paper and project page are released!

πŸ’‘ Highlights

  • πŸ”₯ We first demonstrate that GPT-4V can effectively recognize and describe materials, allowing our model to precisely identifies and aligns materials with the corresponding components of 3D objects.
  • πŸ”₯ We construct a Material Library containing thousands of materials with highly detailed descriptions readily for MLLMs to look up and assign.
  • πŸ”₯ An effective pipeline for texture segmentation, material identification and matching, enabling the high-quality application of materials to 3D assets.

πŸ‘¨β€πŸ’» Todo

  • Evaluation for Existed and Model-Generated Assets (both code & test assets)
  • More Interactive Demos (huggingface, jupyter)
  • Make-it-Real Pipeline Inference Code
  • Highly detailed Material Library annotations (generated by GPT-4V)
  • Paper and Web Demos

πŸ’Ύ Installation

Prepare basic modules for deep learning 3d modeling tool(kaolin), rendering engine(blender), and segmentation model. See details in INSTALL.md.

πŸ“¦ Data Preparation

  1. Annotations: in data/material_lib/annotations folder, include:
    • Highly-detailed descriptions by GPT-4V: offering thorough descriptions of the material’s visual characteristics and rich semantic information.
    • Category-tree: Divided into a hierarchical structure with coarse and fine granularity, it includes over 80 subcategories.
  2. PBR Maps: You can download the complete PBR data collection at Huggingface, or download the data used in our project at OpenXLab (Recommended). (If you have any questions, please refer to issue#5)
  3. Material Images(optinal): You can download the material images file here, to check and visualize the material appearance.
Make_it_Real
└── data
    └── material_lib
        β”œβ”€β”€ annotations
        β”œβ”€β”€ mat_images
        └── pbr_maps
            └── train
                β”œβ”€β”€ Ceremic
                β”œβ”€β”€ Concrete
                β”œβ”€β”€ ...
                └── Wood

⚑ Quick Start

Inference

python main.py --obj_dir <object_dir> --exp_name <unique_exp_name> --api_key <your_own_gpt4_api_key>
  • To ensure proper network connectivity for GPT-4V, add proxy environment settings in main.py (optional). Also, please verify the reachability of your API host.
  • Result visualization (blender engine) is located in the output/refine_output dir. You can compare the result with that in output/ori_output.

Annotation Engine

cd scripts/gpt_anno
python gpt4_query_mat.py

Note: Besides functinoning as annotation engine, you can also use this code (gpt4_query_mat.py) to test the GPT-4V connection simply.

❀️ Acknowledgments

  • MatSynth: a Physically Based Rendering (PBR) materials dataset, which offers extensive high-resolusion tilable pbr maps to look up.
  • TEXTure: Wonderful text-guided texture generation model, and the codebase we built upon.
  • SoM: Draw visual cues on images to facilate GPT-4V query better.
  • Material Palette: Excellent exploration of material extraction and generation, offers good insights and comparable setting.

βœ’οΈ Citation

If you find our work helpful for your research, please consider giving a star ⭐ and citation πŸ“

@misc{fang2024makeitreal,
      title={Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials}, 
      author={Ye Fang and Zeyi Sun and Tong Wu and Jiaqi Wang and Ziwei Liu and Gordon Wetzstein and Dahua Lin},
      year={2024},
      eprint={2404.16829},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}