Prompt caching
Opened this issue · 4 comments
Hello! I was thinking that this beautiful and very useful plugin could benefit from prompt caching, for eligible providers.
As for me, I often send the same big note or several notes to llm.
I may have stated the obvious by seeing the last point of the roadmap.
Are you reusing the same note(s) and generating the same number and types of questions? Or just keeping one of the two the same?
the first one. i often use the same note/notes at least three times, because of the size and the low number of questions i request in output, i send it/them multiple times. although sometimes I change the type of questions I ask (t/f, Mcq..).
I see. I'll add prompt caching in one of the next three releases.
Also, I haven't checked with the other providers yet but at least for OpenAI, prompts should already be cached if you're generating the same questions with the same notes using either of the 4o models. I ran a small test just now and you can see some tokens were cached on the second generation.
Oh great! It was just in relation to OAI's announcement of prompt caching that I had thought of this. Also I always feel that the questions generated by Sonnet-3.5 are better compared to gpt4o.