which llama tokenizer to use?

Question

which llama tokenizer to use?

chatsci opened this issue a year ago · 1 comments

In the preprocessing file, we have tokenizer = AutoTokenizer.from_pretrained('trained_models/llama_tokenizer'). This seems won't get the llama tokenizer from HF. which llama tokenizer we should use? As there are several versions on HF. Thanks.

Answer 1 · 2023-06-23T17:25:27.000Z

Hi, sorry for the confusion of the LLaMA tokenizer. We use the LLaMA model and tokenizer from this link: https://huggingface.co/decapoda-research/llama-7b-hf.