Signatures of all metrics
forrestbao opened this issue · 0 comments
forrestbao commented
I wish the signatures of all metrics can be, in line with those in HF's evaluate
library:
- candidates: list[str] or EvalPieces,
- references or docs, list[str] or EvalPiees
- kwargs** only strings, integers or floats
A challenge here is that models may be repetitively loaded, like here
https://github.com/SigmaWe/DocAsRef_0/blob/4ca7c5f645d826ed9d9c60f81526ea49db209328/mnli/eval.py#L15-L19