Code to reproduce the experiments presented in "On the biological plausibility of orthogonal initialisation for solving gradient instability in deep neural networks", N. Manchev, M. Spratling, 2022, King's College London, https://arxiv.org/abs/2211.08408
- SRNN.py -- Trains a Simple RNN with orthogonal initialisation/penalty on the synthetic problems
- SRNN_MNIST.py -- Trains a Simple RNN with orthogonal initialisation/penalty on the MNIST dataset (sequence of pixes tasks)
- FF_MNIST.py -- Trains a feedforward network with orthogonal initialisation/penalty on the MNIST dataset