BenchMetrics Prob: benchmarking of probabilistic error/loss performance evaluation instruments for binary classification problems
Gürol Canbek (2023). BenchMetrics Prob: Benchmarking of probabilistic error / loss performance evaluation instruments for binary-classi cation problems. International Journal of Machine Learning and Cybernetics. doi: 10.1007/s13042-023-01826-5
🔗 Access the free-access full text here.
This repository provides BenchMetrics Prob probabilistic error/loss instrument calculator and simulation tool for benchmarking the robustness of 31 probabilistic error/loss measures/metrics via five criteria and seven simulation cases proposed in my article above.
The proposed method was tested on 30 probabilistic error instruments given in the table below and LogLoss as a probabilistic loss instrument.
Note: Please, cite my article if you would like to use and/or adapt the tool, simulation cases, methodology, and other materials provided and let me know. Thank you for your interest.
| Subtype | Instrument Name | Abbreviation |
|---|---|---|
| (Raw) | Mean Error | ME |
| Squared | Mean Squared Error | MSE |
| Root Mean Square Error | RMSE | |
| Median Squared Error | MdSE | |
| Sum Squared Error | SSE | |
| Normalized Mean Squared Error (v1-5) | nMSE | |
| Absolute | Mean Absolute Error | MAE |
| Median Absolute Error | MdAE | |
| Maximum Absolute Error | MxAE | |
| Geometric Mean Absolute Error | GMAE | |
| Relative | Mean Relative Absolute Error | MRAE |
| Median Relative Absolute Error | MdRAE | |
| Relative | Geometric Mean Relative Absolute Error | GMRAE |
| Relative Absolute Error | RAE | |
| Relative Squared Error | RSE | |
| Percentage | Mean Percentage Error | MPE |
| Mean Absolute Percentage Error | MAPE | |
| Median Absolute Percentage Error | MdAPE | |
| Root Mean Square Percentage Error | RMSPE | |
| Root Median Square Percentage Error | RMdSPE | |
| Percentage (Symmetric) | Symmetric Mean Absolute Percentage Error | sMAPE |
| Normalized Symmetric Mean Absolute Percentage Error | nsMAPE | |
| Normalized Symmetric Median Absolute Percentage Error | nsMdAPE | |
| Scaled | Mean Absolute Scaled Error | MASE |
| Median Absolute Scaled Error | MdASE | |
| Root Mean Squared Scaled Error | RMSSE |