Can it only analysis roc_auc_score?

[xuzhang5788]

Thank you so much for your util.
If I want to analysis precision-recall or f1, how can I use your util?

[mateuszbuda]

Hi, thanks for your interest in this tool.
You can use any score function (score_fun parameter), including a custom one, that returns a single value given arrays of ground truth values and predictions.
For example:

from sklearn.metrics import f1_score

def custom_f1_score(y_true, y_pred):
    return f1_score(y_true, y_pred, average='weighted')

p, z = stat_util.pvalue(y_true, y_pred1, y_pred2, score_fun=custom_f1_score)

[xuzhang5788]

@mateuszbuda
Thank you so much. It is very helpful.

am a little confused about the error bar with your analysis.

When people show their figures, they always show the error bar. Based on my understanding, because machine learning is stochastic, normally, people run their models for many times, and then collect their results and calculate some statistics parameters such as std, etc. and show them in their figures. How different between your analysis and their analysis? Many thanks.

[mateuszbuda]

This tool lets you:

1. compare performance of two models (also based on some statistics) or
2. compute confidence interval for the performance of your model.

For example, you developed a network and trained it 10 times and then computed mean performance, which was 80%.
Then, you added dropout, trained it again 10 times (with the same train/test split) and computed mean performance, which was 90%.
Now, this tool lets you answer:

• Did adding dropout cause a statistically significant improvement in mean performance and what is the level of significance (p-value)?
• What is the confidence interval of mean performance of your models?

You can also do the same based on a single run of you models and compare them instead of summary statistics like mean.

Error bars are usually for reporting standard deviation which is different from confidence interval.
You can read more about confidence intervals here: wikipedia.org/wiki/Confidence_interval

[xuzhang5788]

@mateuszbuda
Thank you so much for your clear explanation. Using your tools, I will get more confident about my model's skills. Thanks a lot!