pisa-engine/pisa

List Splitting/Clipping

Opened this issue · 0 comments

Describe the solution you'd like
There is a bunch of prior work on splitting postings lists; this allows the high impact list to be traversed (more likely) than the low impact list, and since the low impact lists are typically long, accelerates retrieval. Recent work in EMNLP findings showed that this is highly beneficial to learned sparse retrieval mechanisms. I would like to see if this can be pulled into the PISA engine with perhaps a more elegant design (I currently have a private prototype used in the EMNLP experiments but it's a bit rough for release still). This would then supersede the (currently empty) repo here: https://github.com/JMMackenzie/postings-clipping

Paper: https://aclanthology.org/2022.findings-emnlp.205/

Additional context
TBA - This is a draft issue for now.