This repository is the Virtual Appendix for our ICTIR 2021 paper "Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval". In particular, the repository contains the results file generated by the ColBERT PRF Ranker and ReRanker models as well as the baselines reported in our paper. For each model, as per folder, both TREC 2019 and TREC 2020 res files are included.
The implementation of ColBERT PRF is available from our PyTerrier ColBERT plugin.
Results of the ColBERT PRF models as well as the baselines reported in the paper can be reproduced using PyTerrier:
from pyterrier.measures import *
dataset = pt.get_dataset("trec-deep-learning-passages")
res = pt.io.read_results("/path/to/resfile/BM25.2019.res.gz")
evalMeasuresDict = pt.Utils.evaluate(
res,
dataset.get_qrels("test-2019"),
metrics=[ AP(rel=2)@1000, nDCG@10, RR(rel=2)@10 ]
)
print(evalMeasuresDict)- [Macdonald20]: Craig Macdonald, Nicola Tonellotto. Declarative Experimentation inInformation Retrieval using PyTerrier. Craig Macdonald and Nicola Tonellotto. In Proceedings of ICTIR 2020. https://arxiv.org/abs/2007.14271
- [Wang21]: Xiao Wang, Craig Macdonald, Nicola Tonellotto, Iadh Ounis. Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval. In Proceedings of ICTIR 2021. https://arxiv.org/abs/2106.11251
- [Wang23]: Xiao Wang, Craig Macdonald, Nicola Tonellotto, Iadh Ounis. ColBERT-PRF: Semantic Pseudo-Relevance Feedback for Dense Passage and Document Retrieval. ACM Transactions on the Web. https://dl.acm.org/doi/full/10.1145/3572405