/crowdsourcingTruthfulness

repository for the papers: "Crowdsourcing Truthfulness: The Impact of Judgment Scale and Assessor Bias" and "Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor's Background"

Overview

The Many Dimensions of Truthfulness: Crowdsourcing Misinformation Assessments on a Multidimensional Scale

To Appear.

Can the Crowd Judge Truthfulness? A Longitudinal Study on Recent Misinformation about COVID-19

This repository contains the crowdsourced judgments used in the PAUC paper titled "Can the Crowd Judge Truthfulness? A Longitudinal Study on Recent Misinformation about COVID-19"

Citation

If you use this resource, please cite our paper:

Kevin Roitero, Michael Soprano, Beatrice Portelli, Massimiliano De Luise, Damiano Spina, Vincenzo Della Mea, Giuseppe Serra, Stefano Mizzaro, and Gianluca Demartini. Can the Crowd Judge Truthfulness? A Longitudinal Study on Recent Misinformation about COVID-19. In Personal and Ubiquitous Computing, 2021.

BibTeX

@article{roitero2020infodemic,
author = {Roitero, Kevin and Soprano, Michael and  Portelli, Beatrice and De Luise, Massimiliano and Spina, Damiano and
  Della Mea, Vincenzo and Serra, Giuseppe and Mizzaro, Stefano and Demartini, Gianluca},
title = {Can the Crowd Judge Truthfulness? A Longitudinal Study on Recent Misinformation about COVID-19},
booktitle = {Personal and Ubiquitous Computing},
year={2021}
}

The COVID-19 Infodemic: Can the Crowd Judge Recent Misinformation Objectively?

This repository contains the crowdsourced judgments used in the CIKM'20 full paper titled "The COVID-19 Infodemic: Can the Crowd Judge Recent Misinformation Objectively?"

Citation

If you use this resource, please cite our paper:

Kevin Roitero, Michael Soprano, Beatrice Portelli, Damiano Spina, Vincenzo Della Mea, Giuseppe Serra, Stefano Mizzaro and Gianluca Demartini. 2020. The COVID-19 Infodemic: Can the Crowd Judge Recent Misinformation Objectively?. In CIKM ’20: 29th ACM International Conference on Information and Knowledge Management (CIKM2020), Online. October 19-23, 2020.

BibTeX

@InProceedings{roitero2021longitudinal,
author = {Roitero, Kevin and Soprano, Michael and  Portelli, Beatrice and  Spina, Damiano and
  Della Mea, Vincenzo and   Serra, Giuseppe and   Mizzaro, Stefano and Demartini, Gianluca},
title = {The COVID-19 Infodemic: Can the Crowd Judge Recent Misinformation Objectively?},
booktitle = {Proceedings of CIKM'20},
year={2020}
}

Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor's Background

This repository contains the crowdsourced judgments used in the SIGIR'20 full paper titled "Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor's Background"

SIGIR 2020 Talk

If you want to see our talk at SIGIR 2020 visit this link: SIGIR 2020 Talk Video

Citation

If you use this resource, please cite our paper:

Kevin Roitero, Michael Soprano, Shaoyang Fan, Damiano Spina, Stefano Mizzaro, and Gianluca Demartini. 2020. Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor’s Background.In SIGIR ’20: The 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, July 25–30, 2020, Xi’an, China. ACM.

BibTeX

@InProceedings{roitero2020can,
author = {Roitero, Kevin and Soprano, Michael and Fan, Shaoyang and Spina, Damiano
          and Mizzaro, Stefano and Demartini, Gianluca},
title = {Can The Crowd Identify Misinformation Objectively? 
         The Effects of Judgment Scale and Assessor’s Background},
booktitle = {Proceedings of SIGIR'20},
year={2020}
}

Acknowledgements

This work is partially supported by a Facebook Research award and by an Australian Research Council Discovery Project (DP190102141). We thank Devi Mallal from RMIT ABC Fact Check for facilitating access to the ABC dataset.

Crowdsourcing Truthfulness: The Impact of Judgment Scale and Assessor Bias

This repository contains the crowdsourced judgments used in the ECIR'20 short paper titled "Crowdsourcing Truthfulness: The Impact of Judgment Scale and Assessor Bias", for two two different judgment scales: S6 and S100.

The files also include the collected information about assessors’ background that allowed us to analyse assessment bias.

Citation

If you use this resource, please cite our paper:

La Barbera D., Roitero K., Demartini G., Mizzaro S., Spina D. (2020) Crowdsourcing Truthfulness: The Impact of Judgment Scale and Assessor Bias. In: Jose J. et al. (eds) Advances in Information Retrieval. ECIR 2020. Lecture Notes in Computer Science, vol 12036. Springer, Cham. Best Short Paper Award.

BibTeX

@inproceedings{labarbera2020crowdsourcing, 
   title={{Crowdsourcing Truthfulness: The Impact of Judgment Scale and Assessor Bias}},
   booktitle={Proceedings of ECIR'20},
   author={{La Barbera}, David and Roitero, Kevin and Demartini, Gianluca and Mizzaro, Stefano and Spina, Damiano},
   year={2020}
}

Links of Interest

Acknowledgements

This work is partially supported by an Australian Research Council Discovery Project (DP190102141) and a Facebook Research award.