py-irt

Bayesian IRT models in Python

Overview

This repository includes code for fitting Item Response Theory (IRT) models using variational inference.

At present, the one parameter logistic (1PL) model, aka Rasch model, two parameter logistic model (2PL) and four parameter logistic model (4PL) have been implemented. The user can specify whether vague or hierarchical priors are used. The three-parameter logistic model is in the pipeline and will be added when available.

License

py-irt is licensed under the MIT license.

Installation

py-irt is now available on PyPi!

Pre-reqs

Install PyTorch.
Install Pyro
Install py-irt:

pip install py-irt

git clone https://github.com/nd-ball/py-irt.git
cd py-irt
pip install -r requirements.txt

Usage

Once you install from PyPI, you can use the following command to fit an IRT model on the scored predictions of a dataset. For example, if you were to run py-irt with the 4PL model on the scored predictions of different transformer models on the SQuAD dataset, you'd do this: py-irt train 4pl ~/path/to/dataset/eg/squad.jsonlines /path/to/output/eg/test-4pl/

FAQ

What kind of output should I expect on running the command to train an IRT model?

You should see something like this when you run the command given above:

I tried installing py-irt using pip from PyPI. But when I try to run the command py-irt train 4pl ~/path/to/dataset/eg/squad.jsonlines /path/to/output/eg/test-4pl/, I get an error that says bash: py-irt: command not found. How do I fix this?

The CLI interface was implemented in PyPi version 0.2.1. If you are getting this error try updating py-irt:

pip install --upgrade py-irt

Alternatively, you can install the latest version from github:

git clone https://github.com/nd-ball/py-irt.git
cd py-irt
mv ~/py-irt/py_irt/cli.py ~/py-irt/
python cli.py train 4pl ~/path/to/dataset/eg/squad.jsonlines /path/to/output/eg/test-4pl/

How do I evaluate a trained IRT model?

Model evaluation is coming soon!

Citations

If you use this code, please consider citing the following paper:

@inproceedings{lalor2019emnlp,
  author    = {Lalor, John P and Wu, Hao and Yu, Hong},
  title     = {Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds},
  year      = {2019},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing},
}

Implementation is based on the following paper:

@article{natesan2016bayesian,
  title={Bayesian prior choice in IRT estimation using MCMC and variational Bayes},
  author={Natesan, Prathiba and Nandakumar, Ratna and Minka, Tom and Rubright, Jonathan D},
  journal={Frontiers in psychology},
  volume={7},
  pages={1422},
  year={2016},
  publisher={Frontiers}
}

Contributing

This is research code. Pull requests and issues are welcome!

Questions?

Let me know if you have any requests, bugs, etc.

Email: john.lalor@nd.edu

pk1130/py-irt