SigmaWe/DocAsRef

Signatures of all metrics

forrestbao opened this issue · 0 comments

I wish the signatures of all metrics can be, in line with those in HF's evaluate library:

  1. candidates: list[str] or EvalPieces,
  2. references or docs, list[str] or EvalPiees
  3. kwargs** only strings, integers or floats

A challenge here is that models may be repetitively loaded, like here
https://github.com/SigmaWe/DocAsRef_0/blob/4ca7c5f645d826ed9d9c60f81526ea49db209328/mnli/eval.py#L15-L19