/ComfyUI-Flowty-CRM

This is a custom node that lets you use Convolutional Reconstruction Models right from ComfyUI.

Primary LanguagePythonOtherNOASSERTION

ComfyUI-Flowty-CRM

This is a custom node that lets you use Convolutional Reconstruction Models right from ComfyUI.

CRM is a high-fidelity feed-forward single image-to-3D generative model.

example

This node has been adapted from the official implementation with many improvements that make it easier to use and production ready:

  • Added support for cpu generation (initially could only run on cuda)
  • xformers is no longer required
  • separation of the inference pipeline into 3 parts, which enables it to run on low vram gpus and gives users freedom to alter the inputs at any step
  • automatic input image processing
  • high speed and high quality outputs with materials in glb and obj formats
  • closely integrated with comfyui's internal attention code
  • cleaner code, with less dependencies and less memory usage

Installation:

  • Install ComfyUI
  • Clone this repo into custom_nodes: 
    $ cd ComfyUI/custom_nodes
$ git clone https://github.com/flowtyone/ComfyUI-Flowty-CRM.git
  • Install dependencies: 
    $ cd ComfyUI-Flowty-CRM
$ pip install -r requirements.txt
    if using cuda, install cuda dependencies too: 
    $ pip install -r requirements-cuda.txt
  • Download CRM models and place them in ComfyUI/models/checkpoints
  • Start ComfyUI (or restart)
  • Use the example workflow (cpu | cuda)
  • Note: To run the example workflows you will also need to install ComfyUI_essentials

Acknowledgement:

  • Researchers of CRM: Zhengyi Wang, Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, Jun Zhu
  • three.js comfyui interface: MrForExample
  • snr depth reconstruction: Imagedream

This is a community project from flowt.ai. If you like it, check us out!

flowt.ai logo

Troubleshooting

  • NVdiffrast
    • Windows users with cuda(nvidia) gpus might need some special preparations to make the nvdiffrast module work. Please check the official documentation of nvdiffrast for windows installation instructions.
    • CUDA_HOME environment variable is not set - you will need to set an environment variable that points to your cuda installation path. See this stackoverflow answer.
    • If you're experiencing too many issues trying to install NVdiffrast, consider using the cpu workflow by restarting comfyui with the cpu-only option (much slower).
  • Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
    • This usually happens if you tried to run the cpu workflow but have a cuda gpu. Try to restart comfyui and run only the cuda workflow.
  • Portable ComfyUI
    • Users might need to install the dependencies differently, see here.
  • Not enough VRAM/RAM
    • Using these nodes you should be able to run CRM on GPUs with 8GB of VRAM and above, and at least 16GB of RAM.
    • For GPUs with less than 16GB of VRAM, you might need to split the workflow into 4 parts and run them separately. I've added workflows here for demonstration.