About the tokenizer
yusonghust opened this issue · 0 comments
yusonghust commented
I try to print the bos_token_id , eos_token_id , and pad_token_id for llama-7b tokenizer, and find them are all 0. But when I print the bos_token_id , eos_token_id , and pad_token_id for llama-7b model, I get bos_token_id = 0 eos_token_id = 1 and pad_token_id = -1
Is it reasonable?