pipecat-ai/pipecat

Gemini 2.0 Live Api does not generate answers based on tool use's result

Opened this issue · 0 comments

Description

This is more of a feature request as the current way of doing Gemini 2.0 makes the model not answering based on tool results at all.

Issue description

I am using a RAG system using tool use of the new Gemini 2.0 live multimodal. The issue is i give it a tool that is used to retrieve information using RAG, however when the tool result returns, gemini does not generate a follow up answer based on those results at all (basically a tool use's results returned does not trigger a response ?)

Repro steps

Create a tool that Gemini multimodal live uses, this tool must return a result that's used for gemini to generate a follow-up response
Watch how it does not use the information.

Expected behavior

Gemini uses the tool use' results to answer the previous query.

Actual behavior

Gemini just said 'ok i am going to answer your query' and use the tool, and get the tool result back, and nothing else happens

Logs

Not applicable