meta-llama/llama

No special tokens added in tokenizer

manoja328 opened this issue · 0 comments

manoja328 commented

llama/llama/tokenizer.py

Line 23 in 8fac8be

# reload tokenizer

B_INST, E_INST = "[INST]", "[/INST]"
B_SYS, E_SYS = "<>\n", "\n<>\n\n"

SPECIAL_TAGS = [B_INST, E_INST, "<>", "<>"]
UNSAFE_ERROR = "Error: special tags are not allowed as part of the prompt."