This repository contains the data and resources for the SemEval 2024 Task 1: Semantic Textual Relatedness (STR). For more information, please visit the shared task and competition websites.
Dataset | Languages | Shared Task Starter Kit | Citing This Work
If you use our data, please cite our papers:
@misc{ousidhoum2024semrel2024,
title={SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 14 Languages},
author={Nedjma Ousidhoum and Shamsuddeen Hassan Muhammad and Mohamed Abdalla and Idris Abdulmumin and Ibrahim Said Ahmad and
Sanchit Ahuja and Alham Fikri Aji and Vladimir Araujo and Abinew Ali Ayele and Pavan Baswani and Meriem Beloucif and
Chris Biemann and Sofia Bourhim and Christine De Kock and Genet Shanko Dekebo and
Oumaima Hourrane and Gopichand Kanumolu and Lokesh Madasu and Samuel Rutunda and Manish Shrivastava and
Thamar Solorio and Nirmal Surange and Hailegnaw Getaneh Tilaye and Krishnapriya Vishnubhotla and Genta Winata and
Seid Muhie Yimam and Saif M. Mohammad},
year={2024},
eprint={2402.08638},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
@inproceedings{ousidhoum-etal-2024-semeval,
title = "{S}em{E}val-2024 Task 1: Semantic Textual Relatedness for African and Asian Languages",
author = "Ousidhoum, Nedjma and Muhammad, Shamsuddeen Hassan and Abdalla, Mohamed and Abdulmumin, Idris and
Ahmad,Ibrahim Said and Ahuja, Sanchit and Aji, Alham Fikri and Araujo, Vladimir and Beloucif, Meriem and
De Kock, Christine and Hourrane, Oumaima and Shrivastava, Manish and Solorio, Thamar and Surange, Nirmal and
Vishnubhotla, Krishnapriya and Yimam, Seid Muhie and Mohammad, Saif M.",
booktitle = "Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)",
year = "2024",
publisher = "Association for Computational Linguistics"
}The annotation guidelines are available here. (The PDF named SemRel Annotation guidelines.)
Check the SemRel Baseline Folder for details about the baseline experiment.
The STR dataset is available in the data folder or can be downloaded from Hugging Face.
- For Track A: TrackA folder
- For Track B: TrackB folder
- For Track C: TrackC folder
The STR task focuses on the following 14 languages:
- Afrikaans (afr released)
- Algerian Arabic (arq released)
- Amharic (amh released)
- English (eng released)
- Hausa (hau released)
- Indonesian (ind released)
- Hindi (hin released)
- Kinyarwanda (kin released)
- Marathi (mar released)
- Modern Standard Arabic (arb released)
- Moroccan Arabic (ary released)
- Punjabi (pan released)
- Spanish (esp released)
- Telugu (tel released)
A starter kit is available to help you create a baseline result. You can open the starter kit in a Colab Notebook and run the baseline system. The resultant experiment can be submitted to Codalab to ensure the submission format is clear.
To run the Colab Notebook, click the badge "Open in Colab".
- Simple Co-occurrence Baseline for Semantic Relatedness:
