comet-ml/opik

[FR]: Support for Local Models

Closed this issue · 5 comments

Mr-Moonsilver commented

Willingness to contribute

No. I can't contribute this feature at this time.

Proposal summary

Provide support for local models via Ollama or LM Studio.

Motivation

Evaluating the performance of locally deployed models could add a lot of value. Evaluating the performance of a fine-tune or within a custom workflow would make this tool super valuable for local prototyping and testing.

jverre commented

Hey @Mr-Moonsilver

This is something we are actively looking into, how are you using Ollama today ? Are you using it from the command line or through the Python SDK for example ?

Mr-Moonsilver commented

@jverre that is really exciting indeed! I'm using it almost exclusively through the python sdk. I am hosting models locally, some are fine tuned, and using the ollama API to interact with these models.

  • 👍1
jverre commented

Thanks @Mr-Moonsilver, let me take a look and get back to you

jverre commented

Hey @Mr-Moonsilver
I took a look and found 3 different ways to integration Ollama with Opik ! I've created a new documentation page with more information about these here: https://www.comet.com/docs/opik/tracing/integrations/ollama

Let me know what you think

  • 🎉3
Mr-Moonsilver commented

I think this is amazing. Thank you very much!

  • 👍1