Authors: Ali Akbar Septiandri, Marios Constantinides, Mohammad Tahaei, Daniele Quercia
Nokia Bell Labs, Cambridge, United Kingdom
To appear at ACM FAccT 2023
Studies conducted on Western, Educated, Industrialized, Rich, and Democratic (WEIRD) samples are considered atypical of the world's population and may not accurately represent human behavior. In this study, we aim to quantify the extent to which the ACM FAccT conference, the leading venue in exploring Artificial Intelligence (AI) systems' fairness, accountability, and transparency, relies on WEIRD samples. We collected and analyzed 128 papers published between 2018 and 2022, accounting for 30.8% of the overall proceedings published at FAccT in those years (excluding abstracts, tutorials, and papers without human-subject studies or clear country attribution for the participants). We found that 84% of the analyzed papers were exclusively based on participants from Western countries, particularly exclusively from the U.S. (63%). Only researchers who undertook the effort to collect data about local participants through interviews or surveys added diversity to an otherwise U.S.-centric view of science. Therefore, we suggest that researchers collect data from under-represented populations to obtain an inclusive worldview. To achieve this goal, scientific communities should champion data collection from such populations and enforce transparent reporting of data biases.
Clone the repository to your local machine using Git:
git clone https://github.com/aliakbars/weird-facct.gitInstall Poetry, a dependency manager for Python:
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -Alternatively, you can install Poetry using pip:
pip install poetryNavigate to the project directory and use Poetry to install the required dependencies:
poetry installTo reproduce the results and images in the paper, run the facct.ipynb file.
Files named using the chi* pattern have been extracted from the supplementary materials available here.
Files related to FAccT have been manually extracted or gathered from metadata. Datasets necessary for metric calculations are also provided, as detailed in the table below:
| Symbol | Variable | Formula | Description | File |
|---|---|---|---|---|
| Western | Whether country |
western.csv |
||
| Educated | Mean years of schooling for country |
edu.csv |
||
| Industrialized | Level of industrialization for country |
gdp.csv |
||
| Rich | Wealth of country |
gni.csv |
||
| Democratic | Level of democracy for country |
- |