ValueError: Only binary classification is supported. The type of the target is multiclass.

[nchen909]

Hi Markus,
Thanks for sharing your excellent work!
When I run Calibration - (Temp, Iso, Beta, Hist).ipynb
df_iso = cal_results(IsotonicRegression, PATH, files, {'y_min':0, 'y_max':1}, approach = "single")
I got
resnet110_c10

---

ValueError Traceback (most recent call last)
/tmp/ipykernel_443943/2066161569.py in
----> 1 df_iso = cal_results(IsotonicRegression, PATH, files, {'y_min':0, 'y_max':1}, approach = "single")

~/autodl-tmp/NN_calibration/scripts/calibration/cal_methods.py in cal_results(fn, path, files, m_kwargs, approach)
291 probs_val[idx_nan] = 0
292
--> 293 # Get results for test set
294 error, ece, mce, loss, brier = evaluate(softmax(logits_test), y_test, verbose=True, normalize=False)
295 error2, ece2, mce2, loss2, brier2 = evaluate(probs_test, y_test, verbose=False, normalize=True)

~/autodl-tmp/NN_calibration/scripts/calibration/cal_methods.py in evaluate(probs, y_true, verbose, normalize, bins)
201
202 y_prob_true = np.array([probs[i, idx] for i, idx in enumerate(y_true)]) # Probability of positive class
--> 203 brier = brier_score_loss(y_true=y_true, y_prob=y_prob_true) # Brier Score (MSE)
204
205 if verbose:

~/miniconda3/lib/python3.8/site-packages/sklearn/metrics/_classification.py in brier_score_loss(y_true, y_prob, sample_weight, pos_label)
2690 y_type = type_of_target(y_true, input_name="y_true")
2691 if y_type != "binary":
-> 2692 raise ValueError(
2693 "Only binary classification is supported. The type of the target "
2694 f"is {y_type}."

ValueError: Only binary classification is supported. The type of the target is multiclass.

How can I solve it? Thanks!
Best, nchen909

[nchen909]

conda create -n calibration python=3.7
pip install scikit-learn==0.20.1
pip install betacal
solve the problem.