Allow openai compatible endoints

[shuther]

Would it be possible to make the openai base url an environment variable (as well as the model name) so we could experiment with self hosted llm?

[haonan-li]

We are working on it and will update soon.

[pselvana]

A quick hack is replacing the following in GPTClient.py:

            self.client = OpenAI(api_key=openai_dict["key"])

with:

            self.client = OpenAI(
                api_key="not-needed", base_url="http://localhost:1234/v1"
            )

If you dockerized it and are running on mac then you can use docker.for.mac.localhost instead of localhost.

Hacky Dockerfile in the root folder if that is helpful --- you would have to update the secret_dict.template Serper API value as well:

FROM python:3.9.16-bullseye
RUN apt-get update && apt-get install -y vim
ADD . / OpenFactVerification/
WORKDIR /OpenFactVerification
RUN cp /OpenFactVerification/factcheck/config/secret_dict.template /OpenFactVerification/factcheck/config/secret_dict.py
RUN pip install -r requirements.txt
RUN python -m nltk.downloader punkt

EXPOSE 2024
ENTRYPOINT [ "python", "webapp.py" ]

I'm thinking of connecting an additional evidence crawler to get around needing Serper: https://github.com/searxng/searxng

[shuther]

Yes that is a good start; ideally it is a variable for the docker that we can adapt in the .env file. For the search api, it is definitely something interesting to leverage https://docs.searxng.org/dev/search_api.html

[haonan-li]

The openai compatible endoints is supported in v0.0.2 see #8.
Docs will be updated soon.

[shuther]

so it is possible to switch between openai and anthropic; could we use ollama or anything else (so change the base url?)

[HanXudong]

so it is possible to switch between openai and anthropic; could we use ollama or anything else (so change the base url?)

The short answer is Yes.

1. Export base URL and key

export ANTHROPIC_API_KEY=... # this is required only if you want to replace openai with anthropic
export LOCAL_API_KEY=... # this is required only if you want to use local LLM
export LOCAL_API_URL=... # this is required only if you want to use local LLM

2. Set your client and model name
   For Anthropic:

python -m factcheck --modal string --input "MBZUAI is the first AI university in the world" --model claude-3-opus-20240229 --prompt claude_prompt

For local hosted models

python -m factcheck --modal string --input "MBZUAI is the first AI university in the world" --client local_openai --model anyModel --prompt pathToPrompt.yaml

However, the prompt can be model-specific, especially if we ask the model to output a JSON format response. We have yet to have a chance to support all models, so you will have to adopt your own prompts when using models other than openai.

[shuther]

not sure if it is expected but LOCAL_API_URL needs to finish with a / (slash); it does not need to be the case usually, not sure why.

[shuther]

in the local_openai_client.py, I would recommend to apply 2 changes to support slower calls:
max_requests_per_minute=2,

and

         model=self.model,
            messages=messages,
           # temperature=0.3,
            timeout=90,

[haonan-li]

Thank you for your suggestion and we believe use max_requests_per_minute to do the traffic control is effective.

For personalized model or API, the arguments will be variant a lot. We will leave comment here to note users.