swabhs/open-sesame

How to run the SemEval script on your system's output

brenowca opened this issue · 2 comments

brenowca commented

Hi,

Thank you very much for this contribution! I just read your paper, and I am now trying out the code.

I am wondering how to put the CoNLL output of the frame semantic parser into the SemEval XML format so I can run the script provided by them as you did in your paper.

Thank you in advance for your time,
Breno

brenowca commented

Could you also, please, give me a heads up on how to run your semafor_evaluation.py script for comparing the Semafor parser?

swabhs commented

Hi Breno,

For ease of reading, I convert the FrameNet XML files into a CoNLL-styled format. This repository permits evaluation for argument identification in two different ways:

  1. First, I have replicated the SemEval evaluation in the repository. Given a predicted file in the CoNLL format, you can run
$ python -m sesame.evaluation <predicted-conll-file>

This gives you the final result, as well as a detailed evaluation in argid-eval.log.

  1. Alternatively, when you test the frameID or argID module, a file in frame-elements file format is written to logs/sample-$MODEL/predicted-1.7-$MODEL-test.fes. This file can now be used for evaluation using scripts from SEMAFOR (a wrapper around the original SemEval pl script).

The semafor_evaluation script produces a frame-elements file from a CoNLL file. To run it, use:

$ python -m sesame.semafor_evaluation --e_mode convert_conll_to_fe --fe_file <output-fe-file> --conll_file <input-conll-file>