switch to a different scoring library

Question

switch to a different scoring library

Closed this issue 3 years ago · 4 comments

pytrec-eval is a pain to install because it downloads some code from the web and compiles it. pyterrier has a version of the library that is a wheel with pre-compiled library. I've tested it with Linux and it works great. Need to test with Windows and Mac and then we can switch over. This will also make it easier to run CI on Github.

Answer 1 · 2021-11-29T21:53:32.000Z

I would suggest using ir-measures.
I believe it is more robust than the current pyterrier one.
From what I know, they are also planning on revising that wheel.

Answer 2 · 2021-12-01T17:48:52.000Z

@eugene-yang ir-measures depends on the pyterrier one which is why I chose it since it has fewer dependencies and we don't need the additional functionality provided by ir-measures

Answer 3 · 2021-12-02T19:20:49.000Z

I believe ir-measures only adds cwl-eval>=1.0.10 and pyndeval>=0.0.2 aside from pytrec-eval-terrier.
Will using ir-measures be more robust and easier to support more evaluation metrics in the future?

Answer 4 · 2021-12-02T20:19:01.000Z

pyterrier-eval-terrier does everything we need. It is a drop in replacement for pytrec-eval. In the future, if we need additional metric support, we can switch over to ir-measures.