[DISCUSSION]: Metrics for NNCF

[AbhijithGanesh]

Which metrics would be necessary to benchmark NNCF

I recently came across the need for benchmarking and making Quality/Analytics tool as mentioned in the Wiki.

I would like the experienced folks (@alexsu52 and @andrey-churkin) to mention the metrics that can be integrated into the telemetry as suggested in the Wiki

My key takes

MSE:

• This is by far the de-facto error function calculated for Regression and other similar algorithms

MAE:

• Mean Absolute error is significant needs to be implemented into the telemtry

Classification reports

• I think accuracy, precision, recall, f1-score, etc are necessary to be benchmarked. Vast majority of NeuNets classify and without these metrics, classification is Incomplete

SQNR:

• Another important metric mentioned

[alexsu52]

Hello @AbhijithGanesh,

If you talking about GSoC task then the suggested metrics are good suggestion for estimation per layer quantization error.