Implementation is based on the following references:
- Hinton, G., Osindero, S., & Teh, Y.-W. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18(7), 1527–1554.
- Hinton, G. (2010). A practical guide to training restricted Boltzmann machines. Momentum, 9(1), 926.
- Palm, R. B. (2012). Prediction as a candidate for learning deep hierarchical models of data (Master's thesis).
- Larochelle, H., & Bengio, Y. (2008). Classification using discriminative restricted Boltzmann machines. In Proceedings of the 25th international conference on Machine learning (pp. 536–543). ACM.
I wrote this code as part of my undergraduate project several years ago. It was an attempt to reproduce results from Hinton's 2006 paper "A fast learning algorithm for deep belief nets". Inspired by Geoffrey Hinton's neural network simulation, a visualization of the generative learning process of a DBM, I also attempted to produce a similar sort of visualization. When it came to implementing the softmax units in the top-most "associative memory" layer of the DBN, I found the descriptions in Larochelle 2008 to be very useful - some of which are reproduced below.
The following samples were generated by randomly initializing the visible units in the associative layer and "clamping" the softmax units to one of the target classes. After each iteration of Gibbs sampling, a downward pass through the network was performed to generate a sample, which was saved to an image file. The animations show how the samples evolve over 200 iterations of Gibbs sampling. It's interesting to see how quickly the Gibbs sampling converges.
To model the joint the distribution of input data and associated target classes, the visible layer contains an additional group of softmax units, used to represent the alternative target classes in the MNIST training data.
For the "associative memory" layer of the DBN, the energy function includes additional terms for the parameters of the softmax units, which respectively represent the softmax-to-hidden connections and the softmax biases.
Training the the associate memory layer of the DBN is the same as training an RBM with standard binary units, but the conditional probabilities have a slightly different form.
The conditional probability of a particular hidden unit being active includes additional terms for the softmax units.
To choose the most probable class under the model, the conditional distribution can be computed exactly for reasonable numbers of classes. In practice, the unnormalized probabilities can be used to determine the most probable class.