What is the function of `<END-OF-TURN>` Token?

[mantle2048]

Thanks a lot for this excellent work.

I am confused about the token <END-OF-TURN> . I tested the following example in MiniViewer:

First, I added <END-OF-TURN> after the multi-modal input, but the model refused to answer the question.

However, when I removed <END-OF-TURN>, chameleon was able to provide a reasonable completion.

I want to know why this is happening. What does <END-OF-TURN> mean for Chameleon?

[geyuying]

Same question when running inference with inference/examples/multimodal_input.py. The models often generates "I'm sorry, but I'm unable to comply with that request".

[mantle2048]

Based on my testing experience, canceling <END-OF-TURN> token and increasing temperature will increase the probability of a reasonable completion. But as long as <END-OF-TURN> is attached, the probability of the model rejecting the reply greatly increases.

[jacobkahn]

This is expected behavior based on how the model was tuned. You can increase the likelihood of generating a safe response by adding a more specific query about an image.