Scoring Question

[schbrough]

Greetings,

I am using the BertScore for multilingual translation scoring. I have a source in English, a vendor provided translation, and a Spanish translation produced by Azure's AI Services. When I use the English version as the reference, my two Spanish translations average scores in the 80's. However, if I use a tool like Google translate to take my source doc and make it Spanish, then use that as the reference, my two Spanish translations now average scores in the 90's.

Is there some verbiage I should use to account for the higher scores when the reference text, and candidates are in the same language? Why does the score get marked lower when performing multilingual comparisons if in fact, they are saying the same thing?