- Invented by Claude Shannon
-
Information theory: Shannon discovered these formulas about information transmission. Such as how many bits per second should be able to transfer over various media. Coined the term "bit".
-
Laid ground work to switch from analog to digital communication. The Bell phone systems' analog communications were degraded when sent over long distances and noise was amplified by repeaters. By switching to digital, Bell implemented a type of digital repeater that reduced degradation of signal, reducing loss, and improving communication.
-
Shannon Capacity: the fundamental limit of information that can be sent across a medium
-
Shannon entropy or level of surprise in a message/communication
- A symbolic analysis of relay and switching circuits (https://dspace.mit.edu/handle/1721.1/11173)
- Communication Theory of Secrecy Systems (https://web.archive.org/web/20070605092733/http://netlab.cs.ucla.edu/wiki/files/shannon1949.pdf)
- The Mathematical Theory of Communication (https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf)
-
Self-Information
-
Entropy
-
Mutual Information
-
Conditional Entropy
-
Kullback-Leibler Divergence
-
Binary Cross Entropy
-
Multiclass Cross Entropy