platformatic/ai-warp

Allow the use of multiple providers

Opened this issue · 0 comments

As titled, but my thoughts on how it could work:

Config schema changes

We just allow multiple AI providers in the config,

{
  aiProviders: {
    openai: { /* ... */ },
    mistral: { /* ... */ },
    // ...
  }
}

Request Body Changes

Add an optional provider property, so the body could look something like

{
  prompt: 'asd123',
  provider: 'openai'
}

Api changes

fastify.ai.warp and fastify.ai.warpStream could then take in the provider property from the request's body. The call to build() in warp.ts could return a lookup map (provider name:implementation). We then check that lookup map to see if the provider is enabled. If it is, we call it. Otherwise, throw a 400.

Concerns

Since the provider property in the body is optional, we need to figure out how to decide which provider to default to if we're not given one in the body.

I think we leave this up to the user, adding a defaultProvider option in the config for hardcoding it, e.g.

{
  defaultProvider: 'openai',
  aiProviders: {
    openai: { /* ... */ },
    mistral: { /* ... */ },
    // ...
  }
}

then in the api we have fastify.ai.getDefaultProvider() to dynamically decide what the default provider is. Could be useful for managing rate limits with the provider, ab testing, etc. By default, the function will just return what's set in the config.