Errors when running example script
Opened this issue · 4 comments
When running the example script from the readme, I get the following error:
ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating
I was able to resolve this by manually setting the chat template:
llava_1_5_chat_template = '''{% for message in messages %}
{% if message.role == 'system' %}
{{ message.content.strip() }}
{{ '\n' }}
{% elif message.role == 'user' %}
{% if message.content is string %}
USER: {{ message.content.strip() }}
{% elif message.content is iterable %}
USER:
{% for content in message.content %}
{% if content.type == 'image_url' and content.image_url is mapping and content.image_url.url is string%}
{{ content.image_url.url.strip() + ' ' }}
{% endif %}
{% endfor %}
{% for content in message.content %}
{% if content.type == 'text' %}
{{ content.text.strip() }}
{% endif %}
{% endfor %}
{% endif %}
{{ '\n' }}
{% elif message.role == 'assistant' and message.content is not none %}
ASSISTANT: {{ message.content.strip() }}
{{ '\n' }}
{% endif %}
{% endfor %}
{% if add_generation_prompt %}
ASSISTANT:
{% endif %}'''
model_path = "mlx-community/llava-1.5-7b-4bit"
tokenizer_config = {"chat_template": llava_1_5_chat_template}
model, processor = load(model_path, tokenizer_config)
However, when running generate
, I now get the following error, which I suppose will be fixed in #67.
TypeError: Model.__call__() got an unexpected keyword argument 'image_grid_thw'
Hey @DePasqualeOrg
I managed to replicate the issue and will fix it in #67
Yesterday, I accidentally introduced a bug on v0.0.14 and v0.0.15.
This template issue is being cause by transformers :(
I have to reupload all models again.
They moved the templates to the preprocessor config file.