Training data construction for reader verifier
dsl-light opened this issue · 3 comments
Hello and thanks again!!!
I'm trying to reproduce the reader(SQUAD 2.0 alike) part. If I'm not wrong, the reader is also a path re-ranker to help get the best path that contains the answers and supporting sentences. About this I have 2 questions: (1)How are the negative paths(is impossible=True) constructed? by TF-IDF or the upstream retriever? (2) What if the negative paths contain part of the supporting sentences, or even the answer(eg. for comparison question)? also make is_impossible==True?
Hi, thanks for posting your questions!
(1) How are the negative paths(is impossible=True) constructed?
We use TF-IDF based negative examples. We also use SQuAD 2.0 data as mentioned in Appendix B.3 in our paper.
by TF-IDF or the upstream retriever?
I'm not sure what "upstream retriever" means... We tried to feed negative examples selected by our graph retriever (running our graph retriever on SQuAD open data to simulate the inference time) in our experiments, but we found that it did not give performance improvements, and took some time as we need to run our model on 90+k SQuAD examples. Thus, we do not try those negative examples when we trained our best models.
(2) What if the negative paths contain part of the supporting sentences, or even the answer(eg. for comparison question)? also make is_impossible==True?
We filter out negative paragraphs with answer strings, as sometimes those paragraphs can be equally correct and plausible, but they are just not annotated as supporting facts.
If you want to reproduce the results, it might be easier to use our train data available here.
I close this issue now, but please feel free to reopen or reach me via email if you have any followup questions! :)