zellyn/deeplearning-class-2011

Code for Deep Learning class at Google

Overview

My code for the Deep Learning class exercises. There should be nothing proprietary in here. For the earlier exercises, I tried to create parallel implementations in Octave and NumPy. Later on, class-supplied helper code necessitated the use of Matlab (for now).

Materials

• OpenClassroom Regression Tutorial
• UFLDF Tutorial Wiki

Note

• The L-BFGS Matlab code is licensed by Stanford under a Creative Commons, Attribute, Non-Commercial license. Please read the details on the UFLDL wiki.
• The MNIST digit data comes from http://yann.lecun.com/exdb/mnist/.

Questions

UFLDL

“Since J(W,b) is a non-convex function, gradient descent is susceptible to local optima; however, in practice gradient descent usually works fairly well.” - UFLDL/Backpropagation

Why? Is it almost convex? Are the local optima all of a similar quality? Are any of the variations (squared error / squared error + weight decay / squared error + weight decay + sparsity constraints) convex?

Tasks

Python

Get stackedae_exercise.py to work.

Implement linear_decoder_exercise.

Implement cnn_exercise.