Pinned Repositories
CrowdTruth
Version 1.0 of the CrowdTruth Framework for crowdsourcing ground truth data, for training and evaluation of cognitive computing systems. Check out also version 2.0 at https://github.com/CrowdTruth/CrowdTruth-core. Data collected with CrowdTruth methodology: http://data.crowdtruth.org/. Our papers: http://crowdtruth.org/papers/
CrowdTruth-core
CrowdTruth framework for crowdsourcing ground truth for training & evaluation of AI systems
adversarial-nibbler
This dataset contains results from all rounds of Adversarial Nibbler. This data includes adversarial prompts fed into public generative text2image models and validations for unsafe images. There will be two sets of data: all prompts submitted and all prompts attempted (sent to t2i models but not submitted as unsafe).
cats4ml-dataset
This dataset is a result of the CATS4ML (Crowdsourcing Adverse Test Sets for Machine Learning) Data Challenge - an adversarial test-set sampling images and labels from the Open Images Dataset for state-of-the-art image classification models. The challenge invited participants to sample this existing publicly available dataset for images that are incorrectly classified by image classification models. It was announced at the HCOMP 2020 conference and ran for three months (Jan-Apr 2021) aiming at submissions by researchers and developers worldwide. This challenge is a first proof-of-concept for the approach using an existing AI dataset, and shows immediate positive impact on improving evaluation datasets in AI research. In the following subsections we describe the main components of the challenge pipeline and data used.
dices-dataset
This repository contains two datasets with multi-turn adversarial conversations generated by human agents interacting with a dialog model and rated for safety by two corresponding diverse rater pools.
capturing-bias
DrDetectiveV1
Crowd Truth Dr. Detective Game
meatspace
toxicity
watsonc
Watson-Crowdsourcing
laroyo's Repositories
laroyo/watsonc
Watson-Crowdsourcing
laroyo/DrDetectiveV1
Crowd Truth Dr. Detective Game
laroyo/capturing-bias
laroyo/meatspace
laroyo/toxicity