Automated scoring of written and spoken test responses is a growing field in educational natural language processing. Automated scoring engines employ machine learning models to predict scores for such responses based on features extracted from the text/audio of these responses. Examples of automated scoring engines include Project Essay Grade for written responses and SpeechRater for spoken responses.
Rater Scoring Modeling Tool (RSMTool) is a python package which automates and combines in a single pipeline multiple analyses that are commonly conducted when building and evaluating such scoring models. The output of RSMTool is a comprehensive, customizable HTML statistical report that contains the output of these multiple analyses. While RSMTool does make it really simple to run a set of standard analyses using a single command, it is also fully customizable and allows users to easily exclude unneeded analyses, modify the default analyses, and even include custom analyses in the report.
We expect the primary users of RSMTool to be researchers working on developing new automated scoring engines or on improving existing ones. Note that RSMTool is not a scoring engine by itself but rather a tool for building and evaluating machine learning models that may be used in such engines.
For installation and usage, please see the official documentation.
- Python >=3.6, <3.7
numpyscipyscikit-learnstatsmodelsskllpandasipythonjupyternotebookseaborn
Contributions to RSMTool are very welcome. Please refer to the documentation for how to get started on developing new features or functionality for RSMTool.
If you are using RSMTool in your work, you can cite it as follows:
Madnani, Nitin and Loukina, Anastassia. "RSMTool: A Collection of Tools for Building and Evaluating Automated Scoring Models". Journal of Open Source Software 1(3), 2016.
@article{MadnaniLoukina2016,
doi = {10.21105/joss.00033},
url = {http://dx.doi.org/10.21105/joss.00033},
year = {2016},
month = {jul},
publisher = {The Open Journal},
volume = {1},
number = {3},
author = {Nitin Madnani and Anastassia Loukina},
title = {{RSMTool}: A Collection of Tools for Building and Evaluating Automated Scoring Models},
journal = {{Journal of Open Source Software}}
}See GitHub Releases.