hipe-eval/HIPE-scorer

evaluation NEL with multiple links

Closed this issue · 1 comments

@maud @e-maud @simon-clematide

With c457b7f, I added the two options to evaluate on a multitude of links:

  • option 1: unionize an arbitrary number of columns (e.g. literal and metonymic columns)
  • option 2: ranked list across separated by pipe within single cell. Participants can provide as many links as they want, the number is limited with a script parameter.

There are some limitations:

  • only NEL, not NERC
  • either option 1 or 2, not both
  • alternative links are allowed in the prediction file only, not in the gold standard
  • alternative links always have identical spans regardless of pipe separation or cross-column separation.
  • blaming the first link of the ranked list in case of spurious links

The limations are not due to technical problems but rather to keep it as simple as possible. Our scorer grew in complexity already considerably due to our advanced evaluation scenarios.

Remove unionizing feature as of f23c50c to keep the script as clean and tidy as possible. The same behaviour can be easily achieved with a pre-processing step merging both columns in a ranked list.