Per claim recall - no partial points. 1 if all evidence found. 0 if only partial information

Question

Per claim recall - no partial points. 1 if all evidence found. 0 if only partial information

Closed this issue 6 years ago · 2 comments

Do not give any partial credit if only partial set of documents is returned.

Answer 1 · 2017-12-12T23:39:05.000Z

@andreasvlachos just to confirm:
I am thinking here we could test that at all pages from least one complete set of evidence is found to score a recall point
The other option is to check if all pages from all sets of evidence are found for each claim in order to score a recall point.

I think option 1 would be more similar to how we plan to score the shared task - and would be a simpler task than option 2. What do you think?

Answer 2 · 2017-12-13T09:03:03.000Z

Yes, option 1 is more sensible!

…

On Tue, 12 Dec 2017 at 23:39, James Thorne ***@***.***> wrote: @andreasvlachos <https://github.com/andreasvlachos> just to confirm: I am thinking here we could test that at all pages from least one complete set of evidence is found to score a recall point The other option is to check if all pages from all sets of evidence are found for each claim in order to score a recall point. I think option 1 would be more similar to how we plan to score the shared task - and would be a simpler task than option 2. What do you think? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#21 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABbUhcTcZyxqc-2HGTs-5impDEZCj4dMks5s_w6ZgaJpZM4Q_UTy> .