stanford-futuredata/ARES

ARES as a Chunk Reranker in a RAG app?

naskovai opened this issue · 3 comments

naskovai commented

Currently I'm building a chunk reranker for a RAG app and am looking into ARES as a reliable evaluation framework.

However, since it evaluates 'context relevance', I think that I may be able to use it directly to rank chunks in a Q&A task.

Alternatively if it's too slow I can use high confidence 'context relevance' predictions as chunk relevance labels and train my own reranker on them.

Does any of that make sense?

robbym-dev commented

Hey @naskovai

Could you specify the latency requirements and the domain focus for your reranker to better tailor advice?

naskovai commented

Hey @robbym-dev, the application domain is "chat with files you have access to". Latency requirements are similar to ChatGPT.

robbym-dev commented

Hey @naskovai,

Given your latency requirements similar to ChatGPT and the application domain, it sounds like ARES is fast enough for your needs. However, I would advise running some tests to gauge its performance in your application domain and verify if it meets your quality standard for performance.

Please feel free to let us know if you have any further questions or if we can assist in any other way. Thank you!

Best,
Robby