/ComfyUI-3D-Pack

An extensive node suite that enables ComfyUI to process 3D inputs (Mesh & UV Texture, etc) using cutting edge algorithms (3DGS, NeRF, etc.)

Primary LanguagePythonMIT LicenseMIT

ComfyUI-3D-Pack

Make 3D assets generation in ComfyUI good and convenient as it generates image/video!
This is an extensive node suite that enables ComfyUI to process 3D inputs (Mesh & UV Texture, etc.) using cutting edge algorithms (3DGS, NeRF, etc.) and models (InstantMesh, CRM, TripoSR, etc.)

Features — Install — Roadmap — Development — Tips — Supporters

Install:

Can be installed directly from ComfyUI-Manager🚀

Alternatively you can download Comfy3D-WinPortable made by YanWenKun

  • Pre-builds are available for:
    • Windows 10/11
    • Python 3.12
    • CUDA 12.4
    • torch 2.5.1+cu124
  • install.py will download & install Pre-builds automatically according to your runtime environment, if it couldn't find corresponding Pre-builds, then build script will start automatically, if automatic build doesn't work for you, then please check out Semi-Automatic Build Guide
  • If you have any missing node in any open Comfy3D workflow, try simply click Install Missing Custom Nodes in ComfyUI-Manager
  • In case there is missing python library, you can check all the python dependencies of my dev environment
  • If for some reason your comfy3d can't download pre-trained models automatically, you can always download them manually and put them in to correct folder under Checkpointsdirectory, but please DON'T overwrite any exist .json files
  • Docker install please check DOCKER_INSTRUCTIONS.md
  • Note: at this moment, you'll still need to install Visual Studio Build Tools for windows and install gcc g++ for Linux in order for InstantNGP & Convert 3DGS to Mesh with NeRF and Marching_Cubes nodes to work, since those two nodes used JIT torch cpp extension that builds in runtime, but I plan to replace those nodes

For manual install

# Fetch newest version of Comfy3D
cd Your ComfyUI Root Directory\ComfyUI\custom_nodes\
git clone https://github.com/MrForExample/ComfyUI-3D-Pack.git
cd ComfyUI-3D-Pack

# Install all dependencies
Your ComfyUI Root Directory\python_embeded\python.exe -s -m pip install -r requirements.txt
Your ComfyUI Root Directory\python_embeded\python.exe install.py

Features:

  • For use cases please check out Example Workflows. [Last update: 16/December/2024]

  • TRELLIS: microsoft/TRELLIS

    2024-12-16.20-06-04.mp4

  • Hunyuan3D_V1 tencent/Hunyuan3D-1

    2024-11-08.19-55-16.mp4

  • StableFast3D: Stability-AI/stable-fast-3d

    2024-08-01.19-21-58.mp4

  • CharacterGen: zjp-shadow/CharacterGen

    zQ4Kb8kXr7QBxkY5.mp4

  • Unique3D: AiuniAI/Unique3D

    • Four stages pipeline:
      1. Single image to 4 multi-view images with resolution: 256X256
      2. Consistent Multi-view images Upscale to 512X512, super resolution to 2048X2048
      3. Multi-view images to Normal maps with resolution: 512X512, super resolution to 2048X2048
      4. Multi-view images & Normal maps to 3D mesh with texture
    • To use the All stage Unique3D workflow, Download Models:
      • sdv1.5-pruned-emaonly and put it into Your ComfyUI root directory/ComfyUI/models/checkpoints
      • fine-tuned controlnet-tile and put it into Your ComfyUI root directory/ComfyUI/models/controlnet
      • ip-adapter_sd15 and put it into Your ComfyUI root directory/ComfyUI/models/ipadapter
      • OpenCLIP-ViT-H-14, rename it to OpenCLIP-ViT-H-14.safetensors and put it into Your ComfyUI root directory/ComfyUI/models/clip_vision
      • RealESRGAN_x4plus and put it into Your ComfyUI root directory/ComfyUI/models/upscale_models
    • Model weights: https://huggingface.co/spaces/Wuvin/Unique3D/tree/main/ckpt

    2024-06-0711-51-27-ezgif.com-resize-video.mp4

  • Era3D MVDiffusion Model: pengHTYX/Era3D

    2024-06-0711-56-48-ezgif.com-resize-video.mp4

  • InstantMesh Reconstruction Model: TencentARC/InstantMesh

    2024-05-23.17-35-22.mp4
    2024-05-23.17-32-31.mp4

  • Zero123++: SUDO-AI-3D/zero123plus

    • Single image to 6 view images with resolution: 320X320

  • Convolutional Reconstruction Model: thu-ml/CRM

    • Three stages pipeline:
      1. Single image to 6 view images (Front, Back, Left, Right, Top & Down)
      2. Single image & 6 view images to 6 same views CCMs (Canonical Coordinate Maps)
      3. 6 view images & CCMs to 3D mesh
    • Note: For low vram pc, if you can't fit all three models for each stages into your GPU memory, then you can divide those three stages into different comfy workflow and run them separately
    • Model weights: https://huggingface.co/sudo-ai/zero123plus-v1.2

    2024-03-12.22-05-10.mp4

  • TripoSR: VAST-AI-Research/TripoSR | ComfyUI-Flowty-TripoSR

    2024-03-05.22-38-36.mp4

  • Wonder3D: xxlong0/Wonder3D

    Wonder3D_FatCat_MVs

  • Large Multiview Gaussian Model: 3DTopia/LGM

    2024-02-08.23-36-31.mp4

  • Triplane Gaussian Transformers: VAST-AI-Research/TriplaneGaussian

    2024-02-08.23-57-37.mp4

  • Preview 3DGS and 3D Mesh: 3D Visualization inside ComfyUI:

    2024-02-04.19-20-17.mp4

  • Stack Orbit Camera Poses: Automatically generate all range of camera pose combinations

    • You can use it to conditioning the StableZero123 (You need to Download the checkpoint first), with full range of camera poses in one prompt pass

    • You can use it to generate the orbit camera poses and directly input to other 3D process node (e.g. GaussianSplatting and BakeTextureToMesh)

    • Example usage:


    • Coordinate system:

      • Azimuth: In top view, from angle 0 rotate 360 degree with step -90 you get (0, -90, -180/180, 90, 0), in this case camera rotates clock-wise, vice versa.
      • Elevation: 0 when camera points horizontally forward, pointing down to the ground is negitive angle, vice versa.

  • FlexiCubes: nv-tlabs/FlexiCubes

    • Multi-View depth & mask (optional normal maps) as inputs
    • Export to 3D Mesh
    • Usage guide:
      • voxel_grids_resolution: determine mesh resolution/quality
      • depth_min_distance depth_max_distance : distance from object to camera, object parts in the render that is closer(futher) to camera than depth_min_distance(depth_max_distance) will be rendered with pure white(black) RGB value 1, 1, 1(0, 0, 0)
      • mask_loss_weight: Control the silhouette of reconstrocted 3D mesh
      • depth_loss_weight: Control the shape of reconstrocted 3D mesh, this loss will also affect the mesh deform detail on the surface, so results depends on quality of the depth map
      • normal_loss_weight: Optional. Use to refine the mesh deform detail on the surface
      • sdf_regularizer_weight: Helps to remove floaters in areas of the shape that are not supervised by the application objective, such as internal faces when using image supervision only
      • remove_floaters_weight: This can be increased if you observe artifacts in flat areas
      • cube_stabilizer_weight: This does not have a significant impact during the optimization of a single shape, however it helps to stabilizing training in somecases

    2024-04-12-16-21-24.mp4

  • Instant NGP: nerfacc

    • Multi-View images as inputs
    • Export to 3D Mesh using marching cubes

  • 3D Gaussian Splatting

    • Improved Differential Gaussian Rasterization
    • Better Compactness-based Densification method from Gsgen,
    • Support initialize gaussians from given 3D mesh (Optional)
    • Support mini-batch optimazation
    • Multi-View images as inputs
    • Export to standard 3DGS .ply format supported

  • Gaussian Splatting Orbit Renderer

    • Render 3DGS to images sequences or video, given a 3DGS file and camera poses generated by Stack Orbit Camera Poses node

  • Mesh Orbit Renderer

    • Render 3D mesh to images sequences or video, given a mesh file and camera poses generated by Stack Orbit Camera Poses node

  • Fitting_Mesh_With_Multiview_Images

    • Bake Multi-View images into UVTexture of given 3D mesh using Nvdiffrast, supports:
    • Export to .obj, .ply, .glb

  • Save & Load 3D file

    • .obj, .ply, .glb for 3D Mesh
    • .ply for 3DGS

  • Switch Axis for 3DGS & 3D Mesh

    • Since different algorithms likely use different coordinate system, so the ability to re-mapping the axis of coordinate is crucial for passing generated result between differnt nodes.

  • Customizable system config file

    • Custom clients IP address
    • Add your huggingface user token

Roadmap:

  • Integrate CharacterGen

  • Improve 3DGS/Nerf to Mesh conversion algorithms:

    • Find better methods to converts 3DGS or Points Cloud to Mesh (Normal maps reconstruction maybe?)

  • Add & Improve a few best MVS algorithms (e.g 2DGS, etc.)

  • Add camera pose estimation from raw multi-views images

Development

How to Contribute

  1. Fork the project
  2. Make Improvements/Add New Features
  3. Creating a Pull Request to dev branch

Project Structure

  • nodes.py:
    Contains the interface code for all Comfy3D nodes (i.e. the nodes you can actually seen & use inside ComfyUI), you can add your new nodes here

  • Gen_3D_Modules:
    A folder that contains the code for all generative models/systems (e.g. multi-view diffusion models, 3D reconstruction models). New 3D generative modules should be added here

  • MVs_Algorithms:
    A folder that contains the code for all multi-view stereo algorithms, i.e. algorighms (e.g. Gaussian Splatting, NeRF and FlexiCubes) that takes multi-view images and convert it to 3D representation (e.g. Gaussians, MLP or Mesh). New MVS algorithms should be added here

  • web:
    A folder that contains the files & code (html, js, css) for all browser UI related things (e.g. the html layout, style and the core logics for preview 3D Mesh & Gaussians). New web UI should be added here

  • webserver:
    A folder that contains the code for communicate with browser, i.e. deal with web client requests (e.g. Sending 3D Mesh to client when requested with certain url routes). New web server related functions should be added here

  • Configs:
    A folder that contains different config files for different modules, new config should be added here, use a sub folder if there are more than one config to a single module (e.g. Unique3D, CRM)

  • Checkpoints:
    A folder that contains all the pre-trained model and some of the model architecture config files required by diffusers, New checkpoints if could be downloaded automatically by Load_Diffusers Pipeline node, then it should be added here

  • install.py:
    Main install script, will download & install Pre-builds automatically according to your runtime environment, if it couldn't find corresponding Pre-builds, then build script will start automatically, called by ComfyUI-Manager right after it installed the dependencies listed in requirements.txt using pip
    If the new modules your are trying to add needs some additional packages that cannot be simplly added into requirements.txt and build_config.remote_packages, then you can try to add it by modify this script

  • _Pre_Builds:
    A folder that contains the files & code for build all required dependencies, if you want to pre-build some additional dependencies, then please check _Pre_Builds/README.md for more informations

Tips

Supporters