Allow the use of multiple providers
Opened this issue · 0 comments
As titled, but my thoughts on how it could work:
Config schema changes
We just allow multiple AI providers in the config,
{
aiProviders: {
openai: { /* ... */ },
mistral: { /* ... */ },
// ...
}
}
Request Body Changes
Add an optional provider
property, so the body could look something like
{
prompt: 'asd123',
provider: 'openai'
}
Api changes
fastify.ai.warp
and fastify.ai.warpStream
could then take in the provider
property from the request's body. The call to build()
in warp.ts could return a lookup map (provider name:implementation). We then check that lookup map to see if the provider is enabled. If it is, we call it. Otherwise, throw a 400.
Concerns
Since the provider
property in the body is optional, we need to figure out how to decide which provider to default to if we're not given one in the body.
I think we leave this up to the user, adding a defaultProvider
option in the config for hardcoding it, e.g.
{
defaultProvider: 'openai',
aiProviders: {
openai: { /* ... */ },
mistral: { /* ... */ },
// ...
}
}
then in the api we have fastify.ai.getDefaultProvider()
to dynamically decide what the default provider is. Could be useful for managing rate limits with the provider, ab testing, etc. By default, the function will just return what's set in the config.