zhudotexe/kani

Working in Huggingface

evanbrociner opened this issue · 2 comments

evanbrociner commented

Hi thank you for this great repo. Any chance you can provide any example of the Huggingface engine. I am having trouble getting it to work. Thank you!

zhudotexe commented

Sure - if you're working with LLaMA v2 or Vicuna (or models that use the same prompt format) we have engines implemented for each of these already and you can just pass model_id. If you are looking to implement your own chat model on HuggingFace, you can use the Vicuna engine as an example of how to implement build_prompt and message_len: https://github.com/zhudotexe/kani/blob/main/kani/engines/huggingface/vicuna.py

See also https://kani.readthedocs.io/en/latest/engines.html#huggingface for more information on the HuggingEngine.

zhudotexe commented

Moving this issue to discussions.