Aleafy/Make_it_Real

[NeurIPS 2024] Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials

Make-it-Real

Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials
Ye Fang*, Zeyi Sun*, Tong Wu, Jiaqi Wang, Ziwei Liu, Gordon Wetzstein, Dahua Lin

*Equal Contribution

📜 News

🚀 [2024/6/8] We release our inference pipeline of Make-it-Real, including material matching and generation of albedo-only 3D objects.

🚀 [2024/6/8] Material library annotations generated by GPT-4V and data engine are released!

🚀 [2024/4/26] The paper and project page are released!

💡 Highlights

• 🔥 We first demonstrate that GPT-4V can effectively recognize and describe materials, allowing our model to precisely identifies and aligns materials with the corresponding components of 3D objects.
• 🔥 We construct a Material Library containing thousands of materials with highly detailed descriptions readily for MLLMs to look up and assign.
• 🔥 An effective pipeline for texture segmentation, material identification and matching, enabling the high-quality application of materials to 3D assets.

👨‍💻 Todo

• Evaluation for Existed and Model-Generated Assets (both code & test assets)
• More Interactive Demos (huggingface, jupyter)
• Make-it-Real Pipeline Inference Code
• Highly detailed Material Library annotations (generated by GPT-4V)
• Paper and Web Demos

💾 Installation

Prepare basic modules for deep learning 3d modeling tool(kaolin), rendering engine(blender), and segmentation model. See details in INSTALL.md.

📦 Data Preparation

1. Annotations: in data/material_lib/annotations folder, include:
   • Highly-detailed descriptions by GPT-4V: offering thorough descriptions of the material’s visual characteristics and rich semantic information.
   • Category-tree: Divided into a hierarchical structure with coarse and fine granularity, it includes over 80 subcategories.
2. PBR Maps: You can download the complete PBR data collection at Huggingface, or download the data used in our project at OpenXLab (Recommended). (If you have any questions, please refer to issue#5)
3. Material Images(optinal): You can download the material images file here, to check and visualize the material appearance.

Make_it_Real
└── data
    └── material_lib
        ├── annotations
        ├── mat_images
        └── pbr_maps
            └── train
                ├── Ceremic
                ├── Concrete
                ├── ...
                └── Wood

⚡ Quick Start

Inference

python main.py --obj_dir <object_dir> --exp_name <unique_exp_name> --api_key <your_own_gpt4_api_key>

• To ensure proper network connectivity for GPT-4V, add proxy environment settings in main.py (optional). Also, please verify the reachability of your API host.
• Result visualization (blender engine) is located in the output/refine_output dir. You can compare the result with that in output/ori_output.

Annotation Engine

cd scripts/gpt_anno
python gpt4_query_mat.py

Note: Besides functinoning as annotation engine, you can also use this code (gpt4_query_mat.py) to test the GPT-4V connection simply.

❤️ Acknowledgments

• MatSynth: a Physically Based Rendering (PBR) materials dataset, which offers extensive high-resolusion tilable pbr maps to look up.
• TEXTure: Wonderful text-guided texture generation model, and the codebase we built upon.
• SoM: Draw visual cues on images to facilate GPT-4V query better.
• Material Palette: Excellent exploration of material extraction and generation, offers good insights and comparable setting.

✒️ Citation

If you find our work helpful for your research, please consider giving a star ⭐ and citation 📝

@misc{fang2024makeitreal,
      title={Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials}, 
      author={Ye Fang and Zeyi Sun and Tong Wu and Jiaqi Wang and Ziwei Liu and Gordon Wetzstein and Dahua Lin},
      year={2024},
      eprint={2404.16829},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}