/how-many-crowd-workers

repository for the TOIS paper "How Many Crowd Workers Do I Need? On Statistical Power when Crowdsourcing Relevance Judgments"

how-many-crowd-workers

Repository for the TOIS paper "How Many Crowd Workers Do I Need? On Statistical Power when Crowdsourcing Relevance Judgments".

ACM DL

Paper link: https://dl.acm.org/doi/10.1145/3597201

Data

The data used in this experiment can be found here: https://github.com/KevinRoitero/crowdsourcingTruthfulness

Code

The code used to compute the set sizes is available here: https://github.com/KevinRoitero/set-size-estimation

The code used to implement the genetic algorithm is available here: https://github.com/Miccighel/Crowd-Size-Gen-EA

Citation

@article{10.1145/3597201,
author = {Roitero, Kevin and Barbera, David La and Soprano, Michael and Demartini, Gianluca and Mizzaro, Stefano and Sakai, Tetsuya},
title = {How Many Crowd Workers Do I Need? On Statistical Power When Crowdsourcing Relevance Judgments},
year = {2023},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
issn = {1046-8188},
url = {https://doi.org/10.1145/3597201},
doi = {10.1145/3597201},
note = {Just Accepted},
journal = {ACM Trans. Inf. Syst.},
month = {may},
keywords = {Relevance judgments, Statistical analysis, Crowdsourcing}
}