Wrong precision and recall in entity linking evaluation
creat89 opened this issue · 5 comments
The scorer seems to calculate the precision and recall wrongly due to the use of consecutive IDs as gold standard and predictions.
Let's consider that the following text is the gold standard. There are three mentions and according to the scorer, there are three links Q69345, Q78068340 and Q39:
Neuchâld B-loc O B-loc.adm.reg O O O Q69345 _ LED0.33
SI B-pers O B-pers.ind O B-comp.title O Q78068340 _ NoSpaceAfter|LED0.09
. I-pers O I-pers.ind O I-comp.title O Q78068340 _ LED0.09
la O O O O O O _ _ _
Confédération B-loc O B-loc.adm.nat O O O Q39 _ LED0.00
Suisse I-loc O I-loc.adm.nat O O O Q39 _ NoSpaceAfter|LED0.00
And let us consider the predicted links:
Neuchâld B-loc O B-loc.adm.reg O O O NIL _
SI B-pers O B-pers.ind O B-comp.title O NIL _
. I-pers O I-pers.ind O I-comp.title O NIL _
la O O O O O O _ _
Confédération B-loc O B-loc.adm.nat O O O Q39|Q340787|Q568452 _
Suisse I-loc O I-loc.adm.nat O O O Q39|Q340787|Q568452 _
Although the entity linker predicted two NIL and Q39, the scorer considers that we only predicted one NIL and Q39. Therefore, we have:
P=1/(1+1) = 0.5 Q39/(Q39+NIL)
R=1/(1+2) = 0.33 Q39/(Q39+Q69345+Q78068340)
The error can happens on the other way too. Let us consider the following text as the gold standard:
Neuchâld B-loc O B-loc.adm.reg O O O NIL _ LED0.33
SI B-pers O B-pers.ind O B-comp.title O NIL _ NoSpaceAfter|LED0.09
. I-pers O I-pers.ind O I-comp.title O NIL _ LED0.09
la O O O O O O _ _ _
Confédération B-loc O B-loc.adm.nat O O O Q39 _ LED0.00
Suisse I-loc O I-loc.adm.nat O O O Q39 _ NoSpaceAfter|LED0.00
And this the predicted output:
Neuchâld B-loc O B-loc.adm.reg O O O Q69345 _
SI B-pers O B-pers.ind O B-comp.title O NIL _
. I-pers O I-pers.ind O I-comp.title O NIL _
la O O O O O O _ _
Confédération B-loc O B-loc.adm.nat O O O Q39|Q340787|Q568452 _
Suisse I-loc O I-loc.adm.nat O O O Q39|Q340787|Q568452 _
In this case, the scorer says, that we predicted three links but the gold standard had only two:
P=1/(1+2) = 0.33 Q39/(Q39+NIL+Q78068340)
R=1/(1+1) = 0.5 Q39/(Q39+NIL)
The scorer should take in consideration the NER boundaries to determine when a link starts and ends. This behavior is not seen if we add an extra line without a link, such as in:
Neuchâld B-loc O B-loc.adm.reg O O O NIL _ LED0.33
REMOVEME O O O O O O _ _
SI B-pers O B-pers.ind O B-comp.title O NIL _ NoSpaceAfter|LED0.09
. I-pers O I-pers.ind O I-comp.title O NIL _ LED0.09
la O O O O O O _ _ _
Confédération B-loc O B-loc.adm.nat O O O Q39 _ LED0.00
Suisse I-loc O I-loc.adm.nat O O O Q39 _ NoSpaceAfter|LED0.00
Hello, and thanks for your message.
This is not really a bug but rather a limitation of the annotation scheme. As the entity linking part of the README says
"as there is no IOB-tagging, a consecutive row of identical links is considered as a single entity.”
Without IOB tags on link annotations it is not possible to determine when a new entity/link starts. If the same entity link (e.g. NIL) is predicted consecutively, the scorer cannot know whether it spans across one or more entities. This behaviour should affect consecutive NIL links exclusively since systems assign links on the basis of mentions and it would be very unlikely to have twice the same QID for two consecutive (and different) mentions. Even if, two consecutive and same non-NIL QIDs should not have an impact, but we will check how many consecutive NIL links span across more than one entity in the reference.
Hello again,
I've been thinking on how this could be solved. And I think the main reason of the previous issues, is that the predictions are not being split into spans using the gold standard spans, either defined by NER tags or consecutive QID.
For the following gold standard:
1 Neuchâld B-loc O B-loc.adm.reg O O O Q69345 _ LED0.33
2 SI B-pers O B-pers.ind O B-comp.title O Q78068340 _ NoSpaceAfter|LED0.09
3 . I-pers O I-pers.ind O I-comp.title O Q78068340 _ LED0.09
4 la O O O O O O _ _ _
5 Confédération B-loc O B-loc.adm.nat O O O Q39 _ LED0.00
6 Suisse I-loc O I-loc.adm.nat O O O Q39 _ NoSpaceAfter|LED0.00
7 est O O O O O O _ _ _
You can define that gold standard spans as: [1,1] -> Q69345; [2,3] -> Q78068340; [4, 4] -> _; [5,6] -> Q39; [7,7] -> _. Yes, I am including the _ as a span of size 1 but they would be removed after. This is done to check that the prediction didn't include a QID in a place where it shouldn't.
Then for the predictions:
1 Neuchâld B-loc O B-loc.adm.reg O O O NIL _
2 SI B-pers O B-pers.ind O B-comp.title O NIL _
3 . I-pers O I-pers.ind O I-comp.title O NIL _
4 la O O O O O O Q39|Q340787|Q568452 _
5 Confédération B-loc O B-loc.adm.nat O O O Q39|Q340787|Q568452 _
6 Suisse I-loc O I-loc.adm.nat O O O Q39|Q340787|Q568452 _
7 est O O O O O O _ _
You can split using the gold standard spans to get the predicted QIDs: [1,1] -> NIL; [2,3] -> NIL; [4, 4] -> Q39; [5,6] -> Q39; [7,7] -> _ . Then, based on the spans generate the values for the evaluation and removing the "_" cases, you can have the following data:
Gold standard: Q69345, Q78068340, Q39
Predictions: NIL, NIL, Q39, Q39
In the case one span in the predictions contain more than one QID, you could subsplit them using consecutive QIDs like in:
1 Neuchâld B-loc O B-loc.adm.reg O O O NIL _
2 SI B-pers O B-pers.ind O B-comp.title O NIL _
3 . I-pers O I-pers.ind O I-comp.title O NIL _
4 la O O O O O O Q39|Q340787|Q568452 _
5 Confédération B-loc O B-loc.adm.nat O O O Q39|Q340787|Q568452 _
6 Suisse I-loc O I-loc.adm.nat O O O Q38 _
7 est O O O O O O _ _
Then have: [1,1] -> NIL; [2,3] -> NIL; [4, 4] -> Q39; [5,6] -> Q39,Q38; [7,7] -> _ and in consequence:
Gold standard: Q69345, Q78068340, Q39
Predictions: NIL, NIL, Q39, Q39, Q38
This first error explained in the original post (i.e. consecutive predicted NILs) would be fixed using gold standard spans defined either by consecutive QID or boundaries from NER. The second error (consecutive NILs in the gold standard) would only by fixed if the spans come from the NER boundaries.
In all the cases, I think this evaluation would be more balanced, specially when multiple NILs are predicted.
Hello,
I've seen that you're preparing for HIPE 2022. I would just like to tell you that I applied the modifications some time ago to a fork of the scorer that you can find here:
https://github.com/creat89/CLEF-HIPE-2020-scorer
You can test it and see if the changes would be good to be used in HIPE 2022.
Great, thanks a lot. We are resolving this currently.
this was fixed by the PR, thanks again @creat89 !