zhongpei/Comfyui_image2prompt

Unknown model (vit_so400m_patch14_siglip_384)

Milor123 opened this issue · 1 comments

Milor123 commented

I get this error while try use moondream2 with CUDA, low memory is ON RTX 4070.

Error occurred when executing LoadImage2TextModel:

Unknown model (vit_so400m_patch14_siglip_384)

File "/home/noe/Documentos/ComfyUI/execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/noe/Documentos/ComfyUI/execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/noe/Documentos/ComfyUI/execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/noe/Documentos/ComfyUI/custom_nodes/Comfyui_image2prompt/src/image2text.py", line 48, in get_model
return (Moodream2Model(device=device,low_memory=low_memory),)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/noe/Documentos/ComfyUI/custom_nodes/Comfyui_image2prompt/src/moondream2_model.py", line 41, in __init__
self.model = AutoModelForCausalLM.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/noe/Documentos/ComfyUI/venv/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 556, in from_pretrained
return model_class.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/noe/Documentos/ComfyUI/venv/lib/python3.11/site-packages/transformers/modeling_utils.py", line 3375, in from_pretrained
model = cls(config, *model_args, **model_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/noe/.cache/huggingface/modules/transformers_modules/moondream2/moondream.py", line 15, in __init__
self.vision_encoder = VisionEncoder()
^^^^^^^^^^^^^^^
File "/home/noe/.cache/huggingface/modules/transformers_modules/moondream2/vision_encoder.py", line 98, in __init__
VisualHolder(timm.create_model("vit_so400m_patch14_siglip_384"))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/noe/Documentos/ComfyUI/venv/lib/python3.11/site-packages/timm/models/factory.py", line 67, in create_model

and get this error while try load the bunny-llama3

Hi guys, How are you?
I want report it.

Error occurred when executing LoadImage2TextModel:

You can't pass `load_in_4bit`or `load_in_8bit` as a kwarg when passing `quantization_config` argument at the same time.

File "/home/noe/Documentos/ComfyUI/execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/noe/Documentos/ComfyUI/execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/noe/Documentos/ComfyUI/execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/noe/Documentos/ComfyUI/custom_nodes/Comfyui_image2prompt/src/image2text.py", line 54, in get_model
return (Llama3vModel(device=device,low_memory=low_memory),)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/noe/Documentos/ComfyUI/custom_nodes/Comfyui_image2prompt/src/llama3_model.py", line 67, in __init__
self.model = AutoModelForCausalLM.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/noe/Documentos/ComfyUI/venv/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 556, in from_pretrained
return model_class.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/noe/Documentos/ComfyUI/venv/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2952, in from_pretrained
raise ValueError(

I am not sure why it dont work, it was working for me but i am not sure now about of what happened