Problem about Running Test to get Rouge Score

Question

Problem about Running Test to get Rouge Score

Machine-Tom opened this issue 5 years ago · 13 comments

Hello Yang. First, thank you very much for sharing your method and code. I have met a problem, when doing the model evaluation. Could you please help me figure it out?

I have run this code for testing the model on test set:
python train.py -mode test -bert_data_path ../bert_data/cnndm -model_path MODEL_PATH -visible_gpus -1 -gpu_ranks 0 -batch_size 30000 -log_file LOG_FILE -result_path RESULT_PATH -test_all -block_trigram true -test_from /Users/admin/Desktop/XXX/model_step_50000.pt

An Error is raised:
FileNotFoundError: [Error 2] No such file or directory: 'XXX/.pyrouge/settings.ini'

Have you met this kind of problem before? Or can you provide another ways to calculate ROUGE for the model? Thank you very much!

Answer 1 · 2019-05-28T15:26:40.000Z

I think this problem related to the pyrouge package, so you could try search for solutions for that package. It is kind of because you did not set the rouge home path for pyrouge.

Answer 2 · 2019-05-29T10:53:18.000Z

Thank you for your help. I already got ROUGE.
Here is another problem. To evaluate the model further, I want to compare the prediction on test set and the target text on test. However, when I check the first sample in cnndm.test.5.bert.pt and cnndm_step50000.candidate in result, I found that they are not from the same sample. So, could you please tell me where is the prediction of model on test set?

Answer 3 · 2019-05-29T11:54:19.000Z

it is from the same sample, but it is sorted by input length.

Answer 4 · 2019-05-29T12:30:48.000Z

Oh I got it. Thank you.
I wonna ask two two questions:

Is "cnndm_step50000.candidate" the result generated by model got in 50000th step? Then what is "cnndm_step50000.gold"
As you mentioned in paper, raw data set of CNN/Daily has only abstractive summary, so you generate oracle summaries by greedy algorithm. What is the abstractive summaries you take as reference, when getting the oracle sentence? the "highlight" in CNN/Daily Mail?

Answer 5 · 2019-05-29T17:39:44.000Z

Yes, 50000 is the step. .gold is the gold-standard summary.
Yes, it is the highlight.

Answer 6 · 2019-07-11T17:33:22.000Z

Hi Yang Liu,

Is there a way to get the summary in the same order as in the json or .pt file?

Thank you!

Answer 7 · 2019-08-08T10:49:05.000Z

May I ask how you solved this problem? I also encountered the same situation.

Answer 8 · 2019-08-11T06:32:09.000Z

May I ask how you solved this problem? I also encountered the same situation.

I haven't solved it yet, I put an issue a few days ago

bheinzerling/pyrouge#7

Edit: I figured out the issue. Basically you need to clone both of the pyrouge repos, in different folders, and then point to the pyrouge-1.5.5 in the tools folder. If there's interest I can release a colab notebook for this.

Answer 9 · 2019-08-12T06:02:22.000Z

May I ask how you solved this problem? I also encountered the same situation.

I haven't solved it yet, I put an issue a few days ago

bheinzerling/pyrouge#7

Edit: I figured out the issue. Basically you need to clone both of the pyrouge repos, in different folders, and then point to the pyrouge-1.5.5 in the tools folder. If there's interest I can release a colab notebook for this.

Thank you very much for your reply
I think I have found a solution to this problem. You should install ROUGE first and install pyrouge from GitHub.
The detailed steps can be found in these two blogs.
https://blog.csdn.net/wr339988/article/details/70165090
https://blog.csdn.net/MerryCao/article/details/49174283

Answer 10 · 2020-03-02T21:28:25.000Z

Does anyone have english language instructions?

Answer 11 · 2020-03-25T13:22:36.000Z

Does anyone have english language instructions?

I don't know if this is the same as the contents of the blogs above, but i was able to solve the issue and get it running using the instructions here:
https://stackoverflow.com/questions/45894212/installing-pyrouge-gets-error-in-ubuntu

Answer 12 · 2020-07-24T17:23:06.000Z

May I ask how you solved this problem? I also encountered the same situation.

I haven't solved it yet, I put an issue a few days ago

bheinzerling/pyrouge#7

Edit: I figured out the issue. Basically you need to clone both of the pyrouge repos, in different folders, and then point to the pyrouge-1.5.5 in the tools folder. If there's interest I can release a colab notebook for this.

Do you still have that Colab notebook? I don't really know how to do that.

Answer 13 · 2020-09-21T21:23:07.000Z

Does anyone have english language instructions?

May I ask how you solved this problem? I also encountered the same situation.

I haven't solved it yet, I put an issue a few days ago
bheinzerling/pyrouge#7
Edit: I figured out the issue. Basically you need to clone both of the pyrouge repos, in different folders, and then point to the pyrouge-1.5.5 in the tools folder. If there's interest I can release a colab notebook for this.

Do you still have that Colab notebook? I don't really know how to do that.

I got the same error with PyRouge (ROUGE-1.5.5) during running the PacSum model.
The following steps helped me (see the notebook on Google Colab):

https://colab.research.google.com/drive/1-vAnr3d3W8GtqSCn4MwjrdQrzN0uCXzx?usp=sharing