mindsdb/mindsdb_evaluator

Implement a bunch of accuracy funtions

George3d6 opened this issue · 1 comments

Get AccStats to generate every single viable accuracy metrics it can think of for the problem types.

For starters let's do everything under sklearn's metrics + whatever @paxcema thinks is relevant for timeseries + @hakunanatasha if you have any ideas (or would like to warn about any sklearn metrics being bonk).

As with r2, I think it's better if we start with implementations that are wrappers over the sklearn function, and then add custom behavior if needed for various edge cases.

Also this shouldn't be equivalent to using all of those accuracy function for ensembling, the user should be able to specify a subset of functions that we ensemble on, but every single possible function should be computed by AccStats and added to the model data.

For now we can leave the defaults for ensembling to be the same.

I'm fine with re-evaluating this plan for next quarter, but I do not see a need to implement this now.

While certain metrics from sklearn have undesirable qualities (i.e. negative R2), it is important to maintain performance metrics that are commonly used in case people want to benchmark results against their own solutions, and with techniques that are trusted and well-understood. Re-writing these metrics will require very careful consideration, and strong unit-tests as these are exceptionally important in evaluation criteria.

Are there custom metrics that have actually come up that may require an implementation? Otherwise I would strongly advise against working on this when there are other tasks at hand.