This repository contains a short colab notebook demonstrating the use of a Delta-Orthogonal initialization at criticality to train a deep convolutional neural network.
Link to open and run the notebook in a sandbox environment: Delta-Orthogonal demo
This is based on the paper:
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S. Schoenholz, Jeffrey Pennington
International Conference on Machine Learnig, 2017 arXiv/1806.05393
This is not an official Google product.