Question regarding additional white spaces in Mistral's chat template
Aratako opened this issue · 2 comments
Hello, I am currently using your chat template to adapt Mistral for system messages. Upon inspecting the content, I noticed that in this repository's chat template for Mistral, there are additional white spaces on the assistant side that are not present in the original Mistral-Instruct chat template. Here is the relevant part from the original chat template:
{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token}}
Source: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2/blob/main/tokenizer_config.json
And here is the corresponding part in this repository's chat template:
{% elif message['role'] == 'assistant' %}
{{ ' ' + content.strip() + ' ' + eos_token }}
Source: https://github.com/chujiezheng/chat_templates/blob/main/chat_templates/mistral-instruct.jinja
These differences in the chat template result in outputs that slightly differ from the original Mistral-Instruct after applying tokenizer.apply_chat_template(). For example, the README in this repository shows the following example of output:
<s>[INST] This is a system prompt.
This is the first user input. [/INST] This is the first assistant response. </s>[INST] This is the second user input. [/INST]
Using the original chat template, the output should look like this:
<s>[INST] This is a system prompt.
This is the first user input. [/INST]This is the first assistant response.</s>[INST] This is the second user input. [/INST]
Why have spaces been added after [/INST] and before the eos_token?
Hi, in a fix a few days ago, I removed the second white space as it is a mistake.
For the current fixed version, the template is like:
{% elif message['role'] == 'assistant' %}
{{ ' ' + content.strip() + eos_token }}
The first white space is because I found the generation of Mistral-Instruct usually starts with a white space like Hello, ...
. Since I have applied .strip()
here, the starting white space should be added back.
Oh sorry, I didn't notice you already fixed it! That's my bad.
And I understood the meaning of first whitespace. Thank you for your quick reply!