Identifying individual annotators

[eryk-mazus]

Dear SummEval authors.

Thank you for your work.

I've been exploring the provided expert and crowdsourced annotations and have a question. Does the annotation at the position i in the provided list for each summary always corresponds to the same person ?

[{'coherence': 2, 'consistency': 1, 'fluency': 4, 'relevance': 2},
 {'coherence': 1, 'consistency': 1, 'fluency': 2, 'relevance': 1},
 {'coherence': 1, 'consistency': 1, 'fluency': 3, 'relevance': 2}]

I'd be grateful for answer

[Alex-Fabbri]

Hi,
Thank you for your interest in our work! Yes, the annotation corresponds to the same person for the 4 dimensions.

[eryk-mazus]

Thanks for quick reply