mindsdb/mindsdb_evaluator

Feature Request: Expected calibration error

vijay-jaisankar opened this issue · 2 comments

vijay-jaisankar commented

I would like to raise a PR that implements the ECE metric in this file.

More specifically, the function proposed will take a list of softmax outputs, the y_true labels, and the number of bins. It will then calculate ECE as follows:

ECE calculation source

If this sounds like a good idea, please assign me to the same.

Thanks,
Vijay

paxcema commented

Hey @vijay-jaisankar, love this idea!

Just for reference, one of the pending bits of work from our end is to port into this repo some logic from Lightwood. In particular, we have an ECE implementation here (along with MCE and two other scores). I would recommend that you use it as reference for your contribution.

  • 👍1
vijay-jaisankar commented

Hello @paxcema , thanks for the info, will check it out while making the PR!