Possibility to specify custom API endpoint address?

Question

Possibility to specify custom API endpoint address?

Opened this issue 8 months ago · 4 comments

Could this be implemented?
For example 'localhost:7860/v1' could be one of these custom addresses one could enter as a command line argument for running such tests on local models that are set up with an OpenAI-like endpoint but of course are running off a different address.

Thank you

Elliott

Answer 1 · 2024-04-03T15:31:10.000Z

Hey! Thanks for the note - yep that is definitely doable we just need to add it to the repo. This isn't on our immediate development plans for the maintainers but if you want to throw up a PR we can take a look

Answer 2 · 2024-04-03T19:07:41.000Z

I also need that to be working.
Releasing that would be greatly appreciated.

Answer 3 · 2024-04-03T19:19:16.000Z

https://github.com/vllm-project/vllm/blob/main/examples/openai_chatcompletion_client.py
This vllm client example might worth to look at how they integrated custom url with openai.

Answer 4 · 2024-04-03T20:18:43.000Z

https://github.com/vllm-project/vllm/blob/main/examples/openai_chatcompletion_client.py
This vllm client example might worth to look at how they integrated custom url with openai.

I'm aware of this setting already, unfortunately things are a little more complex as they are using langchain openAI which complicates things a little more than that.