KenyonY/openai-forward

Enable running multiple reverse proxys with different targets.

michaelfeil opened this issue · 2 comments

michaelfeil commented

I have experimented running multiple proxies in the same fastapi-app, forwarding to multiple endpoints.
E.g. /v1/audio could be still forwarded to openai, while /v1/completions is forwarded to this API:
https://github.com/go-skynet/LocalAI

How would you suggest integrating such a feature?

I thought of refactoring ROUTE_PREFIX class OpenaiBase to make openai base configurable.

KenyonY commented

Hi @michaelfeil ,
I understand that your goal is to reverse proxy multiple service addresses (such as api.openai.com and localhost:8080) to the same port. This can be easily achieved by making a few code modifications. For example, you can add another route in openai_forward/app.py:

from sparrow.api import create_app
from .openai import Openai

app = create_app(title="openai_forward", version="1.0")


def add_route(obj: Openai):
    app.add_route(
        obj.ROUTE_PREFIX + "/{api_path:path}",
        obj.reverse_proxy,
        methods=["GET", "POST", "PUT", "DELETE", "OPTIONS", "HEAD", "PATCH", "TRACE"],
    )

openai = Openai()
openai.BASE_URL = "https://api.openai.com"
openai.ROUTE_PREFIX = "/openai"
add_route(openai)

localai = Openai()
localai.BASE_URL = "http://localhost:8080"
localai.ROUTE_PREFIX = "/localai"
add_route(localai)

Then, remove the @classmethod decorator for the _reverse_proxy class method in openai_forward/base.py, otherwise, the modification to BASE_URL will not take effect.

After making these changes, you will be able to access the content of api.openai.com under the /openai route, such as openai/audio/translations, and access all content from the service http://localhost:8080 under the /localai route, such as /localai/v1/completions.

In the future, it's worth considering making ROUTE_PREFIX and BASE_URL configurable for multiple target.

  • 👍1