transitive-bullshit/OpenOpenAI

Add support for custom models

transitive-bullshit opened this issue · 2 comments

transitive-bullshit commented

Currently, the models is hard-coded to use the OpenAI chat completion API, but it wouldn't be very difficult to use custom LLMs or external model providers.

The only real constraint is that the custom models need to support function calling and/or ideally parallel tool calling using OpenAI's tool_calls format.

Will consider implementing this depending on how much love this issue gets.

ChuckJonas commented

Seems like this could be done using one of the function calling models on hugging face.

Most of the work would just revolve around converting the open.chat.completions payload to a prompt that the model likes.

Looking at the example notebook, the format seems to be something like this:

<s> <FUNCTIONS>{
    "function": "search_bing",
    "description": "Search the web for content on Bing. This allows users to search online/the internet/the web for content.",
    "arguments": [
        {
            "name": "query",
            "type": "string",
            "description": "The search query string"
        }
    ]
}</FUNCTIONS>

[INST] Search bing for the tallest mountain in Ireland [/INST]
</s>

Was able to run this using a hugging face endpoint with Llama-2-7b-chat-hf-function-calling-v2, and it seemed to work ok.

praneybehl commented

Possibly add support for Mistral API? Ref: https://docs.mistral.ai/platform/overview/

  • 👍1