Working with ollama or llama.cpp

Question

Working with ollama or llama.cpp

region23 opened this issue a year ago · 10 comments

With the publication of codellama, it became possible to run LLM on a local machine using ollama or llama.cpp.
How to configure your extension to work with local codellama?

flaviodelgrosso commented 10 months ago

+1

Answer 1 · 2023-08-31T09:26:58.000Z

hello @region23

yes it is possible to use a local model. What you'd need to do is:

Serve the model locally at some endpoint
And change the settings accordingly

change it to your local endpoint

and make sure to update the prompt template

Answer 2 · 2023-08-31T15:09:07.000Z

HF Code Error: code - 400; msg - Bad Request

Answer 3 · 2023-08-31T15:30:16.000Z

curl to API is working

Answer 4 · 2023-09-24T10:47:03.000Z

For now ollama's API is not supported, it's on the todo list though!

cf huggingface/llm-ls#17

Answer 5 · 2023-10-11T12:30:43.000Z

Also created an issue for llama.cpp : huggingface/llm-ls#28

Answer 6 · 2023-11-11T01:45:30.000Z

This issue is stale because it has been open for 30 days with no activity.

Answer 7 · 2024-01-17T01:47:20.000Z

This issue is stale because it has been open for 30 days with no activity.

Answer 8 · 2024-02-02T20:16:35.000Z

Is there a timeline for when feat: Add adaptors for ollama and openai #117 might be merged?

Answer 9 · 2024-02-09T17:51:17.000Z

Finishing the last touches of fixes on llm-ls and testing everything works as expected for 0.5.0 and we should be good to go for a release. I'd say either I find some time this week-end or next week :)