About the eos token

[nicosouth]

hi, i have a question about eos token.

In the file(hf_decoder_model.py), i find that you use the class of AutoTokenizer. i want to know if it will add the eos token automatically.

[shizhediao]

Different models have different behaviors on the eos token. If there is a specification of eos token in the model config, then it will add the eos token. Otherwise, the eos token will be None.
You might want to refer to the doc: https://huggingface.co/docs/transformers/v4.32.0/en/model_doc/auto#transformers.AutoTokenizer

Thanks!

[nicosouth]

do you mean it will be defined in the tokenizer_config.json?
I did find that there is a parameter "add_eos_token" in this file.

thank you!

[shizhediao]

yes you can modify it according to your setting.

[nicosouth]

thank you!