The dataset is EMNIST which extends MNIST by including images of handwritten letters (upper and lower case) as well as handwritten digits. Both EMNIST and MNIST are extracted from the same underlying dataset, referred to as NIST Special Database 19. Both use the same conversion process resulting in centred images of dimension 28×28.
- Setting up a baseline system on EMNIST. Initially you need to establish a baseline system on EMNIST using stochastic gradient descent (SGD) and without explicit regularization. Carry out experiments using 100 ReLU hidden units per layer, and investigate using from 2–5 hidden layers. Make sure you use an appropriate learning rate. For the initial experiments, compare systems using the validation set accuracy. Then for the best systems with 2–5 hidden layers compare them using the test set accuracy.
- Implementation and exploration of the RMS-Prop and Adam learning algorithms.
- Implementation and exploration of a learning rate scheduler (cosine annealing) for stochastic gradient descent (SGD) and Adam.
- Implementation and exploration of Adam with regularization and explicit weight decay.