MeteorMetric.evaluate_batch(hyps, refs) fails with error: BrokenPipeError: [Errno 32] Broken pipe

[zakhan4]

Hello,

I have followed the setup steps from the ReadMe. Hence I also have jar file downloaded for Meteor. Then I try to calculate Meteor scores for a batch of summaries and references and get the error Traceback given below.

from summ_eval.meteor_metric import MeteorMetric
def get_meteor(hypothesis, references):
    metric = MeteorMetric()
    return metric.evaluate_batch(hypothesis, references)

Error Traceback:

---------------------------------------------------------------------------
BrokenPipeError                           Traceback (most recent call last)
<ipython-input-11-49bb02aed099> in <module>()
----> 1 get_meteor(mlsum_rnd_sum, mlsum_tgt )

2 frames
<ipython-input-8-cc8dcaf582c5> in get_meteor(hypothesis, references)
     22 def get_meteor(hypothesis, references):
     23     metric = MeteorMetric()
---> 24     return metric.evaluate_batch(hypothesis, references)
/usr/local/lib/python3.7/dist-packages/summ_eval/meteor_metric.py in evaluate_batch(self, summaries, references, aggregate)
    111                 if not isinstance(ref, list):
    112                     ref = [ref]
--> 113                 stat = self._stat(summ, ref)
    114                 eval_line += ' ||| {}'.format(stat)
    115             self.meteor_p.stdin.write(enc('{}\n'.format(eval_line)))

/usr/local/lib/python3.7/dist-packages/summ_eval/meteor_metric.py in _stat(self, hypothesis_str, reference_list)
     79         self.meteor_p.stdin.write(enc(score_line))
     80         self.meteor_p.stdin.write(enc('\n'))
---> 81         self.meteor_p.stdin.flush()
     82         return dec(self.meteor_p.stdout.readline()).strip()
     83 

BrokenPipeError: [Errno 32] Broken pipe

[Alex-Fabbri]

Hi,
I haven't encountered this before, but I would take a look at this issue from nlg-eval (which this meteor code is based off of), which has some tips for debugging errors with the jar file.

[zakhan4]

Hi Alex,

Thanks for responding back. I saw a few days ago that you have created a pip package for SummEval and I tried to install and use that instead to calculate meteor score and it worked. But I forgot to close this issue. So, I'm closing it now!