markus93/NN_calibration

Replacate temperature scaling calibration on a simple CNN model

AnouarITI opened this issue · 2 comments

AnouarITI commented

Hi, I tried to follow your work on NN calibration, and I made my Temperature scaling calibration for my model. Before calibration, I have these metrics:

Accuracy: 71.98
Error: 28.019999999999996
ECE: 0.09495480151921513
MCE: 0.1909070909023285

After calibration, I get these metrics (Which seems ok for my understanding):

Accuracy: 71.98
Error: 28.019999999999996
ECE: 0.028244566865265396
MCE: 0.13784639537334445

However, when I try to plot the reliability diagram, I feel something needs to be fixed!
image

Please help me understand why I am getting this plot for the calibrated model. you can check my notebook if you want.
Calib_NN.zip

markus93 commented

Hey,

thanks for your question. I checked the notebook and noticed that you used validation labels for testing, but the calibrated probabilities are generated on test data.

Instead of that bin_info_cal = get_bin_info(y_confs_cal, y_preds_cal, y_true_val, bin_size=1/bins) should be
bin_info_cal = get_bin_info(y_confs_cal, y_preds_cal, y_test_true, bin_size=1/bins)

Let me know if you have any more questions!

All the best,
Markus Kängsepp

AnouarITI commented

Thank you! The issue is fixed
image

  • 🎉1