Semester-project
In this project we focus on shallow neural networks, only two-layers deep, differently than the common setting in which models are trained on datasets with predefined labels, we focused on the teacher-student framework and assume a teacher network model, also known as oracle, to provide the labels for given instances. The main objective is to utilize numerical experiments and prove theoretical results that can provide some insights on the ”symmetry breaking” phase that happens in the training for this setting, which follows a first phase in which all neurons tend to maximally correlate with the signal. In this work we study the problem using the framework of statistical mechanics, leveraging the concept of order parameters, similarly to previous work of Saad and Solla [1995].