Performance Metrics Package for Imbalanced Data
The package provides MATLAB functions to calculate rank-based and threshold-based performance metrics for imbalanced datasets. Imbalanced datasets are frequently found in many applications. In a typical binary classification problem the imbalance of data can be defined by the skew ratio between the classes:
skew = negative examples / positive examples
In most cases the vast majority of examples are from one class, but the practitioner is typically interested in the minority (positive) class. With a few exceptions, performance scores are attenuated by skewed distributions. Skew is a critical factor in evaluating performance metrics. To avoid or minimize skew-biased estimates of performance, is it possible to normalize the performance scores to a fully balanced set. In these way, classifiers can be compared across databases free of confounds introduced by skew.
The package contains the following performance metrics:
-
Rank-based Metrics:
- Area Under ROC Curve
- Area Under Precision-Recall Curve
- Interpolated Precision
- Precision-Recall Breakeven Point
-
Threshold metrics:
- Accuracy
- Precision
- Recall
- F-Beta scores
- Cohen's Kappa
- Krippendorff's Alpha
- Matthews Correlation Coefficient
For more details on the effect of skew, see
L. A. Jeni, J. F. Cohn and F. De la Torre. 2013. Facing imbalanced data - recommendations for the use of performance metrics. Affective Computing and Intelligent Interaction (ACII 2013) http://www.pitt.edu/~jeffcohn/skew/PID2829477.pdf