Allow for multiple providers in the same application
Opened this issue · 11 comments
Currently each provider (i.e. open-ai, azure open-ai, etc) are in separate extensions that can't co-exist within the same application. It should be feasible to have an application that uses multiple providers.
Think hybrid cloud. Maybe, via configuration, when deployed in Azure the application uses Azure OpenAI, but when deployed on-prem, it uses open-ai. It would be runtime configuration which would "switch" the provider (not build time).
It could also be possible where if one provider is not available/errors out/etc, that the application tries again using an alternate provider.
We have added this at build time
Shouldn't it be at runtime? By having it at build time a user would have to build a new version of their application for different deployment environments, which completely defeats the purpose of being able to have multiple providers.
It should just be a runtime switch, like a datasource url/credentials.
First, I am not convinced that the use described is realistic.
Second, a very good reason to have build time enforcement of this is that it allows for providing good error messages when necessary configuration is missing - because don't forget that each provider needs different configuration
First, I am not convinced that the use described is realistic.
You can't envision that an organization builds an application and deploys multiple instances of the application into different environments? Maybe one instance is deployed into Azure and they configure the application to the Azure OpenAI service where they can burn down their Azure credits/committed spend while also deploying another instance of the application elsewhere (on prem, some other cloud, etc) where they configure the application to go to open-ai?
providing good error messages when necessary configuration is missing - because don't forget that each provider needs different configuration
That can still be done at runtime (I'm doing it in the superheroes).
That can still be done at runtime (I'm doing it in the superheroes)
Not the Quarkus standard way.
What I am trying to explain and explain that the last thing I want to do is add complexity from the get go when no one has a good idea of what patterns are going to be used in practice
You can't envision that an organization builds an application and deploys multiple instances of the application into different environments? Maybe one instance is deployed into Azure and they configure the application to the Azure OpenAI service where they can burn down their Azure credits/committed spend while also deploying another instance of the application elsewhere (on prem, some other cloud, etc) where they configure the application to go to open-ai
This is pure speculation. I want to see people's real needs.
If it turns out that this is one of them, I'll gladly admit you correctly anticipated customer's needs and add the feature
What I am trying to explain and explain that the last thing I want to do is add complexity from the get go when no one has a good idea of what patterns are going to be used in practice
I totally understand that, but I honestly don't think my use case is unrealistic. It also means that for now, as much as I want to, I can not use this extension/feature.
It also means that for now, as much as I want to, I can not use this extension/feature.
That is unfortunate
It is, because this would be a great way to showcase & publicize this work, as well has have an integration point inside a real system that could be used as a test bed.
It seems with this that it has broken out azure open ai & open ai as separate providers/extensions. Maybe they should be combined into a single provider/extension? Under the covers its the same request/response. The only difference is nomenclature and authentication mechanism (azure key vs openai org id/api key). Its like changing a database URL/username/password - something that can/should be done at runtime.
Thought I would reply back here as to what I've done to date in the superheroes with regards to this issue.
- The app has both the openai & azure-openai extensions installed
- I created separate profiles (i.e.
quarkus.profile) and configured lots of the properties accordingly for theopenaiandazure-openaiprofiles - The CI/CD process creates 2 images - 1 for the default profile (
prod) and 1 for theazure-openaiprofile- The "default" profile is
openai, meaning if you want to useazure-openaiyou need to use the separate image
- The "default" profile is
- The build process (unit and integrations tests) will build & test both profiles
See https://github.com/quarkusio/quarkus-super-heroes/tree/main/rest-narration#making-live-calls-to-openai-providers for details.
I've also built my own version of #130. By default, NO calls are made to any openai provider. You have to explicitly turn that on in either case.