jbalma/BoltzmannBaby

BoltzmannBaby is a C/C++ OpenMP-4.0 RBM-based deep learning research code used to understand the underlying thermodynamic properties of deep networks in terms of Temperature, Pressure and Volume -- as well as energy density and entropy currents. It uses general binary input structures by way of binning up (mostly) arbitrary 2-d data structures. Almost any 2-d data can be used as inputs through the use of the binning mechanism. Two binning subroutines are currently available: one for time-series or spatial location data (e.g. images or f(x) style data), and another for character-based data (raw text) in which characters are represented by the y-axis of the matrix, and the spatial location in a sentence or word is represented by the x axis. The current benchmark problems use either the character/text based data, or else a noisy sinusoidal function. The learning rate is automatically adjusted using a simple AdaGrad technique. The bias neurons are updated each epoch or kept fixed depending on user preference. A number of shifted sub-samples can be used to enlarge the data set (useful in text based learning). The default setup uses a set of early childhood reading samples, fables, Kafka short stories to train layer wise each RBM. Arbitrary numbers of additional RBMs can be stacked in the chain, but the default configuration uses eight. Full documentation is in the works.

BoltzmannBaby README

The code for BoltzmannBaby is an experimental and lightweight C/C++ OpenMP-4.0 deep learning code focused on general binary input structures using binning of 2-d data structures. The current test problems use sinusoidal function data, or character/text based data binned into a binary matrix. The learning rate is fixed. The bias neurons are updated each epoch. A number of shifted sub-samples can be used to enlarge the data set. The default setup uses a set of Kafka stories and fables to train a single layer Restricted Boltzmann Machine (RBM). Arbitrary numbers of additional RBMs can be stacked, each with varying topologies.

Data used for the character benchmark is in test.txt

Data used for the sinusoidal binning test is generated on the fly.

To Build

Edit Makefile

Set CC=gcc/icc Set CFLAGS appropriately (-O3)

To Contribute

Do following to update the project here on github in the master branch

git add main.cpp git commit -m "comment on whatever you did" git push -u origin master

Notes

Edit main.cpp number of neurons, epoch length, etc.

Changing K_MAX and the initial sample (V, Vs, Vp) produces interesting effects on learning rate

Current test.txt is big (~1500 64-char lines)