This project contains a TensorFlow implementation of the Ladder Network. The model is written conform the Estimator API, an easy to use interface provided by TensorFlow.
Ladder Networks are a structure of neural networks that allow to be trained with few labeled data. In other words, Ladder Networks provide a way to do semi-supervised learning with deep neural networks. Given a feedforward network the Ladder Network is constructed by creating the inverse network and requiring to reconstruct the input from the output similar to autoencoders.
The concept of Ladder Networks can be used in all kinds of feedforward neural networks or even recurrent networks. However, in this project only fully connected, convolutional, meanpool and maxpool variants are implemented. Each layer of the encoder part (the feedforward network) consists of four parts:
- noise addition layer
- transformation (fc/conv/pool) layer
- batch normalization (cfr. S Ioffe, C Szegedy)
- activation
As the decoder is the encoders counterpart, it consists equivalently of four parts:
- denoise layer (for intuitive explanation: Learning by Denoising
- inverse transformation(fc/deconv/unpool)
- batch normalization
- activation
The order of layers and their size fully parametrized. This makes the Estimator fully customable! However, the convolutional layers are automatically padded with zeros to prevent deconvolution artifacts Deconvolution and Checkerboard Artifacts - A Odena, V Dumoulin, C Olah
A nice intro on Ladder Networks is available on YouTube:
If you want to dive deeper into the subject I suggest you to read:
- Semi-Supervised Learning with Ladder Networks- A Rasmus, M Berglund, M Honkala, H Valpola, T Raiko
- Lateral connections in denoising autoencoders support supervised learning - A Rasmus, H Valpola, T Raiko
- Denoising autoencoder with modulated lateral connections learns invariant representations of natural images - A Rasmus, T Raiko, H Valpola