qaeval could not been imported

[yfqiu98]

Hello I just try to use the QAEval in sacrerouge, however I found the following issue. Seems the loading of QAEval() is broken and I did not see the output as in the demo colab https://colab.research.google.com/drive/1IZUZCUQAdCfXAMDDyeF67QFHB72y2svI?usp=sharing:

"Some weights of BartForConditionalGeneration were not initialized from the model checkpoint at facebook/bart-large and are newly initialized: ['final_logits_bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference."

Version information:
!pip install sacrerouge==0.2.2
!pip install qaeval==0.0.9

I follow the instruction in the colab demo to do !sacrerouge setup-metric qa-eval, but it did not work for downloading the answering model file (Although I got the same output as in the colab demo, there is no 'vocab.txt'). Thus I manually download it from:https://drive.google.com/file/d/1q2Z3FPP9AYNz0RJKHMlaweNhmLQoyPA8/view?usp=sharing.

Type "help", "copyright", "credits" or "license" for more information.
import json
from sacrerouge.metrics import QAEval
qaeval = QAEval()
cores = qaeval.score("asd", ["sadfsdf"])
Traceback (most recent call last):
File "", line 1, in
File "/X/miniconda3/envs/sacrerouge/lib/python3.6/site-packages/sacrerouge/metrics/metric.py", line 77, in score
return super().score(summary, references, **kwargs)
File "/X/miniconda3/envs/sacrerouge/lib/python3.6/site-packages/sacrerouge/metrics/metric.py", line 48, in score
return self.score_all([summary], *args, **kwargs)[0]
File "/X/miniconda3/envs/sacrerouge/lib/python3.6/site-packages/sacrerouge/metrics/metric.py", line 83, in score_all
return super().score_all(summaries, references_list, **kwargs)
File "/X/miniconda3/envs/sacrerouge/lib/python3.6/site-packages/sacrerouge/metrics/metric.py", line 56, in score_all
metrics_lists = self.score_multi_all(summaries_list, *args, **kwargs)
File "/X/miniconda3/envs/sacrerouge/lib/python3.6/site-packages/sacrerouge/metrics/qaeval.py", line 40, in score_multi_all
f'Package "qaeval" could not be imported. Please install "qaeval" before running the metric. '
Exception: Package "qaeval" could not be imported. Please install "qaeval" before running the metric. The minimum required version is 0.0.8

Other attempts:

metric = QAEval(cuda_device=-1)
Traceback (most recent call last):
File "", line 1, in
TypeError: init() got an unexpected keyword argument 'cuda_device'
print(QAEval())
<sacrerouge.metrics.qaeval.QAEval object at 0x7f2c224e5048>

Thanks in advance for any help.

[yfqiu98]

Update:

solved by adding sys.path.append('/X/.sacrerouge')

[danieldeutsch]

I'm glad that you solved it, but I don't actually understand how adding that directory to your path fixed it because there is no Python code which is loaded from there.

If you don't add that to the path, but just try running import qaeval, what happens? That import failed in SacreROUGE, which caused the error that you see. If import qaeval fails, then it will show an exception for why it couldn't be imported.