ValueError in meteor.py on running example

Question

ValueError in meteor.py on running example

Closed this issue 6 years ago · 31 comments

Hi, I'm getting the following error upon running the CLI example:

$ nlg-eval --hypothesis=examples/hyp.txt --references=examples/ref1.txt --references=examples/ref2.txt
Bleu_1: 0.550000
Bleu_2: 0.428174
Bleu_3: 0.284043
Bleu_4: 0.201143
Traceback (most recent call last):
  File "/home/dan/anaconda3/bin/nlg-eval", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/media/dan/hdd/projects/nlg-eval/bin/nlg-eval", line 20, in <module>
    compute_metrics()
  File "/home/dan/anaconda3/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/dan/anaconda3/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/dan/anaconda3/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/dan/anaconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/media/dan/hdd/projects/nlg-eval/bin/nlg-eval", line 16, in compute_metrics
    nlgeval.compute_metrics(hypothesis, references, no_overlap, no_skipthoughts, no_glove)
  File "/media/dan/hdd/projects/nlg-eval/nlgeval/__init__.py", line 34, in compute_metrics
    score, scores = scorer.compute_score(refs, hyps)
  File "/media/dan/hdd/projects/nlg-eval/nlgeval/pycocoevalcap/meteor/meteor.py", line 50, in compute_score
    scores.append(float(dec(self.meteor_p.stdout.readline().strip())))
ValueError: could not convert string to float:

I checked the string in question, and it seems that self.meteor_p.stdout.readline() is returning an empty string.

Can you help me figure out how to address this issue?

I've installed the dependencies and ran the setup script. Also, my java version is 1.8.0_171.

Thanks

Answer 1 · 2018-07-17T13:29:52.000Z

Hey @danlou sorry you're having problems. It seems like you set up everything right so I suspect that there's a problem running the meteor .jar. We haven't gotten that error so if you can test out some stuff for us that would be great. Do you see any other errors in the output?

Can you try running python -m unittest from the repo root (the folder containing setup.py)?

Answer 2 · 2018-07-17T14:34:28.000Z

Sure, thanks for helping. The example didn't give any more output.
Here's the full output from the unittest.


➜  nlg-eval git:(master) ✗ python -m unittest
/home/dan/anaconda3/lib/python3.6/importlib/_bootstrap.py:205: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/home/dan/anaconda3/lib/python3.6/importlib/_bootstrap.py:205: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/home/dan/anaconda3/lib/python3.6/importlib/_bootstrap.py:205: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/home/dan/anaconda3/lib/python3.6/importlib/_bootstrap.py:205: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/home/dan/anaconda3/lib/python3.6/importlib/_bootstrap.py:205: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.
/home/dan/anaconda3/lib/python3.6/importlib/_bootstrap.py:205: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/home/dan/anaconda3/lib/python3.6/site-packages/theano/tests/breakpoint.py:3: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
/home/dan/anaconda3/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /home/dan/anaconda3/lib/python3.6/site-packages/sphinxcontrib: missing __init__
  _warnings.warn(msg.format(portions[0]), ImportWarning)
/home/dan/anaconda3/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /home/dan/anaconda3/lib/python3.6/site-packages/google: missing __init__
  _warnings.warn(msg.format(portions[0]), ImportWarning)
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/importlib/_bootstrap.py:205: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py:274: DeprecationWarning: SO is deprecated, use EXT_SUFFIX
  dist_suffix = distutils.sysconfig.get_config_var("SO")
/home/dan/anaconda3/lib/python3.6/importlib/_bootstrap.py:205: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/home/dan/anaconda3/lib/python3.6/site-packages/scipy/sparse/sparsetools.py:21: DeprecationWarning: `scipy.sparse.sparsetools` is deprecated!
scipy.sparse.sparsetools is a private module for scipy.sparse, and should not be used.
  _deprecated()
EBleu_1: 0.550000
Bleu_2: 0.428174
Bleu_3: 0.284043
Bleu_4: 0.201143
E/home/dan/anaconda3/lib/python3.6/unittest/case.py:629: ResourceWarning: unclosed file <_io.BufferedReader name=11>
  outcome.errors.clear()
/home/dan/anaconda3/lib/python3.6/unittest/case.py:629: ResourceWarning: unclosed file <_io.BufferedReader name=13>
  outcome.errors.clear()
E
======================================================================
ERROR: test_compute_individual_metrics (nlgeval.tests.test_nlgeval.TestNlgEval)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/media/dan/hdd/projects/nlg-eval/nlgeval/tests/test_nlgeval.py", line 16, in test_compute_individual_metrics
    hyp="this is a good test")
  File "/media/dan/hdd/projects/nlg-eval/nlgeval/__init__.py", line 187, in compute_individual_metrics
    score, scores = scorer.compute_score(refs, hyps)
  File "/media/dan/hdd/projects/nlg-eval/nlgeval/pycocoevalcap/meteor/meteor.py", line 50, in compute_score
    scores.append(float(dec(self.meteor_p.stdout.readline().strip())))
ValueError: could not convert string to float: 

======================================================================
ERROR: test_compute_metrics (nlgeval.tests.test_nlgeval.TestNlgEval)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/media/dan/hdd/projects/nlg-eval/nlgeval/tests/test_nlgeval.py", line 35, in test_compute_metrics
    scores = nlgeval.compute_metrics(hypothesis, references)
  File "/media/dan/hdd/projects/nlg-eval/nlgeval/__init__.py", line 34, in compute_metrics
    score, scores = scorer.compute_score(refs, hyps)
  File "/media/dan/hdd/projects/nlg-eval/nlgeval/pycocoevalcap/meteor/meteor.py", line 50, in compute_score
    scores.append(float(dec(self.meteor_p.stdout.readline().strip())))
ValueError: could not convert string to float: 

======================================================================
ERROR: test_compute_metrics_oo (nlgeval.tests.test_nlgeval.TestNlgEval)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/media/dan/hdd/projects/nlg-eval/nlgeval/tests/test_nlgeval.py", line 62, in test_compute_metrics_oo
    "this is sentence2 which has been generated by your model"
  File "/media/dan/hdd/projects/nlg-eval/nlgeval/__init__.py", line 224, in compute_metrics
    score, scores = scorer.compute_score(refs, hyps)
  File "/media/dan/hdd/projects/nlg-eval/nlgeval/pycocoevalcap/meteor/meteor.py", line 44, in compute_score
    stat = self._stat(res[i][0], gts[i])
  File "/media/dan/hdd/projects/nlg-eval/nlgeval/pycocoevalcap/meteor/meteor.py", line 64, in _stat
    self.meteor_p.stdin.flush()
BrokenPipeError: [Errno 32] Broken pipe

----------------------------------------------------------------------
Ran 3 tests in 96.455s

FAILED (errors=3)
Exception ignored in: <bound method Meteor.__del__ of <nlgeval.pycocoevalcap.meteor.meteor.Meteor object at 0x7f64f4e88828>>
Traceback (most recent call last):
  File "/media/dan/hdd/projects/nlg-eval/nlgeval/pycocoevalcap/meteor/meteor.py", line 87, in __del__
BrokenPipeError: [Errno 32] Broken pipe
/home/dan/anaconda3/lib/python3.6/subprocess.py:761: ResourceWarning: subprocess 21621 is still running
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name=6>
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name=8>

Answer 3 · 2018-07-17T15:23:13.000Z

Okay that's not too helpful. Let's try calling the meteor .jar manually. Run:

java -jar -Xmx2G nlgeval/pycocoevalcap/meteor/meteor-1.5.jar - - -stdio -l en -norm

You shouldn't see anything happen. Type in: SCORE ||| test ||| test, then press Enter.
Wait a few seconds (the first call is sometimes slow). You should see:
1.0 1.0 0.0 0.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 1.0 1.0.
Here's how it looks for me:

$ java -jar -Xmx2G nlgeval/pycocoevalcap/meteor/meteor-1.5.jar - - -stdio -l en -norm
SCORE ||| test ||| test
1.0 1.0 0.0 0.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 1.0 1.0

Answer 4 · 2018-07-17T15:28:45.000Z

That seems to work fine.

➜  nlg-eval git:(master) ✗ java -jar -Xmx2G nlgeval/pycocoevalcap/meteor/meteor-1.5.jar - - -stdio -l en -norm
SCORE ||| test ||| test  
1.0 1.0 0.0 0.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 1.0 1.0

Answer 5 · 2018-07-17T15:39:50.000Z

Can you add print statement on line 47 of meteor.py so that we can see what eval_line is and enc('{}\n'.format(eval_line))?
Then we can try those manually with the meteor .jar as well.

Answer 6 · 2018-07-17T15:48:36.000Z

Sure, here it is. Maybe it's a parsing error?

➜  nlg-eval git:(master) ✗ nlg-eval --hypothesis=examples/hyp.txt --references=examples/ref1.txt --references=examples/ref2.txt
Bleu_1: 0.550000
Bleu_2: 0.428174
Bleu_3: 0.284043
Bleu_4: 0.201143
eval line: EVAL ||| 10.0 7.0 4.0 4.0 1.0 1.0 2.0 2.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 3.0 3.0 ||| 10.0 13.0 6.0 7.0 4.0 4.0 4.0 4.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 1.0 2.0 10.0 9.0
enc eval line: b'EVAL ||| 10.0 7.0 4.0 4.0 1.0 1.0 2.0 2.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 3.0 3.0 ||| 10.0 13.0 6.0 7.0 4.0 4.0 4.0 4.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 1.0 2.0 10.0 9.0\n'
Traceback (most recent call last):
  File "/home/dan/anaconda3/bin/nlg-eval", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/media/dan/hdd/projects/nlg-eval/bin/nlg-eval", line 20, in <module>
    compute_metrics()
  File "/home/dan/anaconda3/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/dan/anaconda3/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/dan/anaconda3/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/dan/anaconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/media/dan/hdd/projects/nlg-eval/bin/nlg-eval", line 16, in compute_metrics
    nlgeval.compute_metrics(hypothesis, references, no_overlap, no_skipthoughts, no_glove)
  File "/media/dan/hdd/projects/nlg-eval/nlgeval/__init__.py", line 34, in compute_metrics
    score, scores = scorer.compute_score(refs, hyps)
  File "/media/dan/hdd/projects/nlg-eval/nlgeval/pycocoevalcap/meteor/meteor.py", line 53, in compute_score
    scores.append(float(dec(self.meteor_p.stdout.readline().strip())))
ValueError: could not convert string to float:

Answer 7 · 2018-07-17T15:48:51.000Z

I printed them on my machine and got:

$ nlg-eval --hypothesis=examples/hyp.txt --references=examples/ref1.txt --references=examples/ref2.txt
...
Bleu_3: 0.284043
Bleu_4: 0.201143
EVAL ||| 10.0 7.0 4.0 4.0 1.0 1.0 2.0 2.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 3.0 3.0 ||| 10.0 13.0 6.0 7.0 4.0 4.0 4.0 4.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 1.0 2.0 10.0 9.0
b'EVAL ||| 10.0 7.0 4.0 4.0 1.0 1.0 2.0 2.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 3.0 3.0 ||| 10.0 13.0 6.0 7.0 4.0 4.0 4.0 4.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 1.0 2.0 10.0 9.0\n'
METEOR: 0.295797
ROUGE_L: 0.522104
...

Manually trying it:

$ java -jar -Xmx2G nlgeval/pycocoevalcap/meteor/meteor-1.5.jar - - -stdio -l en -norm
EVAL ||| 10.0 7.0 4.0 4.0 1.0 1.0 2.0 2.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 3.0 3.0 ||| 10.0 13.0 6.0 7.0 4.0 4.0 4.0 4.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 1.0 2.0 10.0 9.0
0.1556568826170604
0.39012536521249613
0.2957971080379661

Answer 8 · 2018-07-17T15:52:08.000Z

Yeah, it does seem strange that there's a difference... According to the prints, it seems that it's returning the same output. Which python version are you running? I'm on Python 3.6.1 |Anaconda custom (64-bit).

Answer 9 · 2018-07-17T16:29:27.000Z

Python 3.6.5 :: Anaconda, Inc.
I doubt that's the issue. It seems like you're using a Mac. We've mainly testing this on Linux and msys on Windows.

Answer 10 · 2018-07-17T16:40:12.000Z

I'm using Ubuntu 16.04. I can't really think of any other differences. I'll try to have a better look at the code tomorrow, and if nothing solves it, I'll try upgrading my version of Python. In the meantime, if you think of something else to test, please let me know (incl. if someone is running my version of Python without issues). Thanks for the help!

…

On Tue, Jul 17, 2018 at 5:29 PM Justin Harris ***@***.***> wrote: Python 3.6.5 :: Anaconda, Inc. I doubt that's the issue. It seems like you're using a Mac. We've mainly testing this on Linux and msys on Windows. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#32 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AQBizI6N8YlXGxl5f7rWQTO7N1M28xa0ks5uHhDogaJpZM4VR272> .

Answer 11 · 2018-07-17T16:44:51.000Z

Hey @danlou, can you please check which locale you're using?

Answer 12 · 2018-07-17T16:47:36.000Z

➜  nlg-eval git:(master) ✗ locale
LANG=en_US.UTF-8
LANGUAGE=en_US
LC_CTYPE=pt_PT.UTF-8
LC_NUMERIC=pt_PT.UTF-8
LC_TIME=pt_PT.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY=pt_PT.UTF-8
LC_MESSAGES="en_US.UTF-8"
LC_PAPER=pt_PT.UTF-8
LC_NAME=pt_PT.UTF-8
LC_ADDRESS=pt_PT.UTF-8
LC_TELEPHONE=pt_PT.UTF-8
LC_MEASUREMENT=pt_PT.UTF-8
LC_IDENTIFICATION=pt_PT.UTF-8
LC_ALL=

Maybe I need to set LC_ALL ?

Answer 13 · 2018-07-17T16:54:00.000Z

You can use the locale command.

Also let's see all the output from Meteor.
Add

            import sys
            print("stderr:")
            while True:
                o = self.meteor_p.stderr.read(1)
                if not o:
                    break
                sys.stdout.write(dec(o))
                sys.stdout.flush()
            print("stdout:")
            while True:
                o = self.meteor_p.stdout.read(1)
                if not o:
                    break
                sys.stdout.write(dec(o))
                sys.stdout.flush()

on line 48 of meteor.py (after self.meteor_p.stdin.flush())

EDIT: Writing using: sys.stdout.write(dec(o)).
On my machine is just hangs because there's no stderr.

Answer 14 · 2018-07-17T16:59:13.000Z

➜  nlg-eval git:(master) ✗ nlg-eval --hypothesis=examples/hyp.txt --references=examples/ref1.txt --references=examples/ref2.txt
Bleu_1: 0.550000
Bleu_2: 0.428174
Bleu_3: 0.284043
Bleu_4: 0.201143
stderr:
Traceback (most recent call last):
  File "/home/dan/anaconda3/bin/nlg-eval", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/media/dan/hdd/projects/nlg-eval/bin/nlg-eval", line 20, in <module>
    compute_metrics()
  File "/home/dan/anaconda3/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/dan/anaconda3/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/dan/anaconda3/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/dan/anaconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/media/dan/hdd/projects/nlg-eval/bin/nlg-eval", line 16, in compute_metrics
    nlgeval.compute_metrics(hypothesis, references, no_overlap, no_skipthoughts, no_glove)
  File "/media/dan/hdd/projects/nlg-eval/nlgeval/__init__.py", line 34, in compute_metrics
    score, scores = scorer.compute_score(refs, hyps)
  File "/media/dan/hdd/projects/nlg-eval/nlgeval/pycocoevalcap/meteor/meteor.py", line 56, in compute_score
    sys.stdout.write(o)
TypeError: write() argument must be str, not bytes

Answer 15 · 2018-07-17T17:03:22.000Z

@danlou Here's my locale output:

LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Answer 16 · 2018-07-17T17:04:47.000Z

@danlou that's good! It means there's stderr. Sorry I changed the line but forgot to change it for you. Change it to: sys.stdout.write(dec(o))

Answer 17 · 2018-07-17T17:07:16.000Z

Nice, we now have some more info to go on:

➜  nlg-eval git:(master) ✗ nlg-eval --hypothesis=examples/hyp.txt --references=examples/ref1.txt --references=examples/ref2.txt
Bleu_1: 0.550000
Bleu_2: 0.428174
Bleu_3: 0.284043
Bleu_4: 0.201143
stderr:
Exception in thread "main" java.util.InputMismatchException
	at java.util.Scanner.throwFor(Scanner.java:864)
	at java.util.Scanner.next(Scanner.java:1485)
	at java.util.Scanner.nextDouble(Scanner.java:2413)
	at edu.cmu.meteor.scorer.MeteorStats.<init>(Unknown Source)
	at Meteor.scoreStdio(Unknown Source)
	at Meteor.main(Unknown Source)
stdout:
Traceback (most recent call last):
  File "/home/dan/anaconda3/bin/nlg-eval", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/media/dan/hdd/projects/nlg-eval/bin/nlg-eval", line 20, in <module>
    compute_metrics()
  File "/home/dan/anaconda3/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/dan/anaconda3/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/dan/anaconda3/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/dan/anaconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/media/dan/hdd/projects/nlg-eval/bin/nlg-eval", line 16, in compute_metrics
    nlgeval.compute_metrics(hypothesis, references, no_overlap, no_skipthoughts, no_glove)
  File "/media/dan/hdd/projects/nlg-eval/nlgeval/__init__.py", line 34, in compute_metrics
    score, scores = scorer.compute_score(refs, hyps)
  File "/media/dan/hdd/projects/nlg-eval/nlgeval/pycocoevalcap/meteor/meteor.py", line 67, in compute_score
    scores.append(float(dec(self.meteor_p.stdout.readline().strip())))
ValueError: could not convert string to float:

Answer 18 · 2018-07-17T17:18:36.000Z

Aha! Maybe your system wants a comma instead of a dot for decimal separation!
I see your locale output is different for LC_NUMERIC.
Maybe you can look into changing it to be like mine?

Sorry, I'm not sure which part is at fault here and if we can even correct this by changing our Python code.

Answer 19 · 2018-07-17T18:39:20.000Z

Presumably Meteor should be calling useLocale(Locale.ROOT) or something similar on their Scanner before trying to parse numbers that are not necessarily in the user's native locale.

Answer 20 · 2018-07-17T18:43:59.000Z

We could maybe work around it in https://github.com/Maluuba/nlg-eval/blob/master/nlgeval/pycocoevalcap/meteor/meteor.py by setting LC_ALL=C in the environment when invoking Java.

Answer 21 · 2018-07-19T16:49:39.000Z

Sorry for the delay, just to report that I've tried changing the LC_NUMERIC but that didn't fix it

➜  nlg-eval git:(master) ✗ export LC_NUMERIC="en_US.UTF-8"
➜  nlg-eval git:(master) ✗ locale
LANG=en_US.UTF-8
LANGUAGE=en_US
LC_CTYPE=pt_PT.UTF-8
LC_NUMERIC=en_US.UTF-8
LC_TIME=pt_PT.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY=pt_PT.UTF-8
LC_MESSAGES="en_US.UTF-8"
LC_PAPER=pt_PT.UTF-8
LC_NAME=pt_PT.UTF-8
LC_ADDRESS=pt_PT.UTF-8
LC_TELEPHONE=pt_PT.UTF-8
LC_MEASUREMENT=pt_PT.UTF-8
LC_IDENTIFICATION=pt_PT.UTF-8
LC_ALL=
➜  nlg-eval git:(master) ✗ nlg-eval --hypothesis=examples/hyp.txt --references=examples/ref1.txt --references=examples/ref2.txt
Bleu_1: 0.550000
Bleu_2: 0.428174
Bleu_3: 0.284043
Bleu_4: 0.201143
Traceback (most recent call last):
  File "/home/dan/anaconda3/bin/nlg-eval", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/media/dan/hdd/projects/nlg-eval/bin/nlg-eval", line 20, in <module>
    compute_metrics()
  File "/home/dan/anaconda3/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/dan/anaconda3/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/dan/anaconda3/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/dan/anaconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/media/dan/hdd/projects/nlg-eval/bin/nlg-eval", line 16, in compute_metrics
    nlgeval.compute_metrics(hypothesis, references, no_overlap, no_skipthoughts, no_glove)
  File "/media/dan/hdd/projects/nlg-eval/nlgeval/__init__.py", line 34, in compute_metrics
    score, scores = scorer.compute_score(refs, hyps)
  File "/media/dan/hdd/projects/nlg-eval/nlgeval/pycocoevalcap/meteor/meteor.py", line 67, in compute_score
    scores.append(float(dec(self.meteor_p.stdout.readline().strip())))
ValueError: could not convert string to float:

Answer 22 · 2018-07-19T16:55:27.000Z

I'm not sure Java respects the separate locale environment variables, I think there's just one locale. Does LC_ALL=C work?

Answer 23 · 2018-07-19T16:56:48.000Z

I trying changing all locales and that did it! LC_ALL=C raised some errors.


➜  nlg-eval git:(master) ✗ export LC_ALL="en_US.UTF-8"
➜  nlg-eval git:(master) ✗ locale
LANG=en_US.UTF-8
LANGUAGE=en_US
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8
➜  nlg-eval git:(master) ✗ nlg-eval --hypothesis=examples/hyp.txt --references=examples/ref1.txt --references=examples/ref2.txt
Bleu_1: 0.550000
Bleu_2: 0.428174
Bleu_3: 0.284043
Bleu_4: 0.201143
METEOR: 0.295797
ROUGE_L: 0.522104
CIDEr: 1.242192
WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.
SkipThoughtsCosineSimilairty: 0.626149
EmbeddingAverageCosineSimilairty: 0.884690
VectorExtremaCosineSimilarity: 0.568696
GreedyMatchingScore: 0.784205
Exception ignored in: <bound method Meteor.__del__ of <nlgeval.pycocoevalcap.meteor.meteor.Meteor object at 0x7fd3cec64fd0>>
Traceback (most recent call last):
  File "/media/dan/hdd/projects/nlg-eval/nlgeval/pycocoevalcap/meteor/meteor.py", line 105, in __del__
  File "/home/dan/anaconda3/lib/python3.6/subprocess.py", line 1572, in kill
AttributeError: 'NoneType' object has no attribute 'SIGKILL'

(my machine is busy and I had to interrupt midway, but it seems to be fixed)

Answer 24 · 2018-07-19T18:46:49.000Z

@danlou thanks again for pointing this out and your patience in debugging. Would you just do one more thing for us and see if the commit I just pushed helps you? You shouldn't need to set the locale in your environment anymore. Just try run from my branch: meteor-locale.

Answer 25 · 2018-07-19T21:49:02.000Z

No problem, I'm glad to help and really appreciate your commitment.

I have no locale problems on that branch :) .
However... that same AttributeError has come up again, and now my machine was free (I erroneously assumed earlier that it crashed for being under heavy load).

➜  nlg-eval git:(meteor-locale) nlg-eval --hypothesis=examples/hyp.txt --references=examples/ref1.txt --references=examples/ref2.txt
Bleu_1: 0.550000
Bleu_2: 0.428174
Bleu_3: 0.284043
Bleu_4: 0.201143
METEOR: 0.295797
ROUGE_L: 0.522104
CIDEr: 1.242192
WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.
SkipThoughtsCosineSimilairty: 0.626149
EmbeddingAverageCosineSimilairty: 0.884690
VectorExtremaCosineSimilarity: 0.568696
GreedyMatchingScore: 0.784205
Exception ignored in: <bound method Meteor.__del__ of <nlgeval.pycocoevalcap.meteor.meteor.Meteor object at 0x7f6381006358>>
Traceback (most recent call last):
  File "/media/dan/hdd/projects/nlg-eval/nlgeval/pycocoevalcap/meteor/meteor.py", line 91, in __del__
  File "/home/dan/anaconda3/lib/python3.6/subprocess.py", line 1572, in kill
AttributeError: 'NoneType' object has no attribute 'SIGKILL'
➜  nlg-eval git:(meteor-locale) locale
LANG=en_US.UTF-8
LANGUAGE=en_US
LC_CTYPE=pt_PT.UTF-8
LC_NUMERIC=pt_PT.UTF-8
LC_TIME=pt_PT.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY=pt_PT.UTF-8
LC_MESSAGES="en_US.UTF-8"
LC_PAPER=pt_PT.UTF-8
LC_NAME=pt_PT.UTF-8
LC_ADDRESS=pt_PT.UTF-8
LC_TELEPHONE=pt_PT.UTF-8
LC_MEASUREMENT=pt_PT.UTF-8
LC_IDENTIFICATION=pt_PT.UTF-8
LC_ALL=

This error doesn't seem to come up when using the python api, so for my use case this actually isn't a problem. Still, if you want any help debugging this, let me know.

In [1]: %paste
from nlgeval import compute_metrics
metrics_dict = compute_metrics(hypothesis='examples/hyp.txt',
                               references=['examples/ref1.txt', 'examples/ref2.txt'])

## -- End pasted text --
Bleu_1: 0.550000
Bleu_2: 0.428174
Bleu_3: 0.284043
Bleu_4: 0.201143
METEOR: 0.295797
ROUGE_L: 0.522104
CIDEr: 1.242192
WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.
SkipThoughtsCosineSimilairty: 0.626149
EmbeddingAverageCosineSimilairty: 0.884690
VectorExtremaCosineSimilarity: 0.568696
GreedyMatchingScore: 0.784205

Answer 26 · 2018-07-19T23:24:16.000Z

Great!

That error is fine, it happens in the destructor of the Meteor class so it's only once you're done processing (notice how it happens in __del__). We tried some things to get rid of it but didn't come up with a nice solution yet.

Answer 27 · 2020-06-10T17:27:19.000Z

Hi, this issue still appears on Windows with Java running on a (e.g.) German system interpreting ',' and '.' exactly the opposite as in English. Wouldn't it be reasonable to override the system locale when invoking Java to run meteor.jar, say by adding the following additional command line arguments in meteor.py:
'-Duser.language=en', '-Duser.country=US',
Unlike the previous fix (#33), this would work for Java regardless of the operating system (see also https://stackoverflow.com/a/8809137/11198424).

Answer 28 · 2020-06-10T17:32:34.000Z

Thank you for using nlg-eval and thank you even more for finding an existing issue instead of making a new one!

That's odd that you're still getting this error. This issue shouldn't happen anymore with the latest master branch since we did other things to set the locale a long time ago. You can try those parameters and let us know how they go.

Answer 29 · 2020-06-10T17:43:08.000Z

Thank you for using nlg-eval and thank you even more for finding an existing issue instead of making a new one!

That's odd that you're still getting this error. This issue shouldn't happen anymore with the latest master branch since we did other things to set the locale a long time ago. You can try those parameters and let us know how they go.

Thanks for the really fast reply :). I forgot to add some details to my post, it's updated now. Specifically, it seems the LC_ALL variable is without effect on Windows. Therefore I think it would be best to use Java's platform-independent command line options to set the locale (see my updated post above).

Answer 30 · 2020-06-10T17:52:56.000Z

Ah that makes sense. I don't think I personally tested this much since I didn't want to mess around with my locales too much. So yes, you're right that this isn't ideal for Windows. I can suggest a few ways that you could get this working quickly in easiest to hardest:

Run it in Git Bash (you should already have Git Bash if you have Git installed on Windows), you should be able to use Python mostly like how you normally do)
Try out those variable changes that you suggested and if they work we can merge in a change
Run it in WSL or a Docker container (you will have to set up your Python environment again)

Answer 31 · 2021-02-01T13:23:04.000Z

Hi, this issue still appears on Windows with Java running on a (e.g.) German system interpreting ',' and '.' exactly the opposite as in English. Wouldn't it be reasonable to override the system locale when invoking Java to run meteor.jar, say by adding the following additional command line arguments in meteor.py:
'-Duser.language=en', '-Duser.country=US',
Unlike the previous fix (#33), this would work for Java regardless of the operating system (see also https://stackoverflow.com/a/8809137/11198424).

I can confirm that this resolved the issue that I was having with a different codebase using Meteor. Thanks! Hopefully others will find this useful as well.