c0sogi/llama-api

Support for ExLlama V2

Closed this issue · 2 comments

c0sogi commented

Exllama v2 seems to be working now. Would you like to test this out?
Simply add version=2 to ExllamaModel as below:

your_gptq_model = ExllamaModel(
    version=2,
    model_path="TheBloke/MythoMax-L2-13B-GPTQ",  # automatic download
    max_total_tokens=4096,
)

Thank you!