platformatic/ai-warp

add test for chat history

mcollina opened this issue · 2 comments

mcollina commented
add test for chat history
flakey5 commented

Been trying to think of ways to get this done. The way I'd imagine this working is we check in the AI provider mocks whether or not the chat history was sent. The issue is then how we could reliably get the result.

A global variable for each provider mock (e.g. export let chatHistoryProvided = false) wouldn't work since the mocks are global and there may be multiple tests testing the same provider at the same time, so I don't think it could guarantee reliability.

We could refactor it to have each test get their own mock but I'm not too sure of 1) how that would work with Undici and 2) how we tell the providers' api clients what fetch to use in AI Warp.

The next best thing I can think of is overriding the fastify.ai.warp and fastify.ai.warpStream functions in the test's AI Warp instance to check whether or not chat history was received, e.g.

// in the test
const [app, port] = await buildAiWarpApp({
  aiProvider: {
    openai: {
      model: 'gpt-3.5-turbo',
      apiKey: ''
    }
  }
})
const warp = app.ai.warp
let receivedChatHistory = false
app.ai.warp = async (req, prompt, chatHistory) => {
  if (chatHistory !== undefined) {
    receivedChatHistory = true
  }
  return warp(req, prompt, chatHistory)
}

// ... continue with test

This won't guarantee the chat history is sent to the provider, however it will at least guarantee we received it in the body. Wdyt?

mcollina commented

A global variable for each provider mock (e.g. export let chatHistoryProvided = false) wouldn't work since the mocks are global and there may be multiple tests testing the same provider at the same time, so I don't think it could guarantee reliability.

Only one test runs at any given time inside a process. The mocks should be reset correctly after each test run.

  • 👍1