This project is dedicated to the exploration and extension of the recent research results published by Schwartz-Ziv and Tishby, which involve the application of the Information Bottleneck principle in Machine Learning, specifically Deep Learning.
The recent surge in applying machine learning (ML) techniques to many-body systems has equipped a hard-working physicist with new tools to tackle long-standing problems in the study of phase transitions, efficient wavefunction representation and reconstruction, and even as a bridge to implement the holographic duality in practice. In the process, some interesting connections between fundamental physical concepts and ML have been built. However, the major caveat of ML is the lack of interpretability. Driven by commercial applications, the focus has mostly been on improving results rather than understanding the underlying algorithms. Advancements in ML are usually achieved by trial-and-error rather than fundamental understanding of mechanisms of the intrinsic working principles in the networks. The application of ML tools in physics, however, demands the obtained predictions to be comprehensive.
One of the pioneers working on the theory of ML is Prof. Naftali Tishby. In his work, widely known as the “Information Bottleneck” principle, he applies concepts from information theory and statistical physics, and demonstrates that these allow us to obtain profound insights into the working principle of ML. Taking up Tishby’s work, we analyze ML on a fundamental level and use the acquired knowledge to contribute to establishing a solid theory of ML, which is expected to lead to the improvement of existing algorithms.
[1] "Opening the Black Box of Deep Neural Networks via Information" (2017), https://arxiv.org/abs/1703.00810
[2] "The information bottleneck method" (2000), https://arxiv.org/abs/physics/0004057
[3] "Information Theory of Deep Learning" [Berlin talk] (2017), https://www.youtube.com/watch?v=bLqJHjXihK8
[4] "Information Theory of Deep Learning" [Moscow talk] (2017), https://www.youtube.com/watch?v=FSfN2K3tnJU