pltrdy/rouge

ROUGE-L sets exclusive=True

Closed this issue · 7 comments

matthiasgalle commented

rouge_l_summary_level sets exclusive=True (https://github.com/pltrdy/rouge/blob/master/rouge/rouge_score.py#L371)

Is there a reason for doing so? In particular as exclusive=False is the default option

pltrdy commented

First, really sorry for not replying before, I just missed the notification.

To be honest I remember comparing ROUGE scores with other libraries (like files2rouge wich uses the "official" ROUGE-1.5.5), so I would say that I found ROUGE-L to be measure in term of unique occurences (exclusive=True) but not ROUGE-n. It does sound weird to me as I write it, we would need to check it.

Can you/have you tried to experimentally (in)validate it?

matthiasgalle commented

thanks Paul. We realized this after seeing that in some experiments RL > R1 which shouldn't happen normally

pltrdy commented

Is it getting better without exclusive=True?

matthiasgalle commented

oh yes. I don't think there is a bug.
I was just wondering why the default options are different in RL and Rn

pltrdy commented

@matthiasgalle I meant, did you remove https://github.com/pltrdy/rouge/blob/master/rouge/rouge_score.py#L371 to make it work?

It understood your comment "RL > R1 which shouldn't happen" as "there is a bug"

matthiasgalle commented

indeed, that is what I did.

If exclusive is set to the same value then everything is fine. I just wondered if there was a specific reason why this is hardcoded in rouge_l_summary_level

pltrdy commented

Alright, I'll fix that then, thanks.