A question about Shannon entropy

[joe3154]

Why should minimize Shannon entropy？The performance is better, the Shannon entropy is bigger?

[mboudiaf]

Why should minimize Shannon entropy ?

Shannon Entropy is used in applications where you have access to unlabelled data. Assuming the classifier you have is already good enough, mininmizing the entropy on unlabelled predictions helps the classifier "refine" its boundaries (by forcing him to have low-entropy, i.e. high confidence predictions on unlabelled data). Please refer to our full explanation in the paper https://arxiv.org/abs/2012.06166.

The performance is better, the Shannon entropy is bigger ?

Not necessarily, Shannon Entropy does not tell you if your classifier is good or bad, it simply tells you if it's predictions are confident or not. In the case where the initial classifier is good already, minimizing the entropy will generally improve perfs, but that's not a golden rule. If the initial classifier is bad, chances are that minimizing the entropy will just make things worse in terms of performances.

[joe3154]

Thanks!

[joe3154]

这是来自QQ邮箱的假期自动回复邮件。   您好，我最近正在休假中，无法亲自回复您的邮件。我将在假期结束后，尽快给您回复。