DaDaCheng/DNN_Spring

A spring–block theory of feature learning in deep neural networks

This repository contains the source code and videos for the paper: A spring–block theory of feature learning in deep neural networks. More details can be found here.

Amusing vidoes of folding ruler expermenets and DNN training in different regimes

When slow and steady pulling:

ruler_lazy.mp4

and the corresponding lazy training of a non-linear DNN:

dnn_lazy.mp4

---

When quick and jerky pulling:

ruler_active.mp4

and the corresponding active training of a non-linear DNN:

dnn_active.mp4

Reproducing phase diagrams in real neural network

./bash/test_pd_ns_nl.sh   # data noise
./bash/test_pd_lr_nl.sh   # learning rate
./bash/test_pd_dp_nl.sh   # drop out
./bash/test_pd_bm_nl.sh   # batch size

Spring block simulation

In the first notebook, we solve the first-order overdamping system by discrete iteration.

In the second notebook, we solve the second-order stochastic differential equation using sdeint.