/MLExam

Primary LanguageJupyter Notebook

A simple framework to define and train Artificial Neural Networks using:

  • Stochastic Gradient descent
  • BFGS
  • L-BFGS

The project hierarchy is as follows:

  • Data/
    • MLCup/ contains the ML-Cup datasets
    • monk_datasets/ contains the MONK datasets
  • src/
    • neural_network.py
      • defines the neural network and the methods train_SGD(), train_BFGS and train_LBFGS().
    • layer.py
      • defines a layer of a neural network.
    • neuron.py
      • defines an abstract neuron of a neural network, and concrete neurons such as InputNeuron, LinearNeuron, SigmoidNeuron, TanHNeuron and ReLuNeuron, each implementing the corresponding activation_function() and activation_function_derivative().
    • loss_functions.py
      • defines loss functions, and their derivatives, such as SquaredError() and EuclideanError()
    • grid_search.py
      • defines methods to perform hyperparameters' grid search
    • validation.py
      • defines methods for validation purposes
    • utils.py
      • defines utility methods for plotting curves ecc.

The experiments are performed in the following Python notebooks under src/:

  • experiments_monk.ipynb
  • experiments_cup.ipynb
  • convexity_study.ipynb
  • bfgs_vs_lbfgs_line_search.ipynb
  • bfgs_vs_lbfgs_m.ipynb
  • grid_search_example.ipynb
  • save_load_network_parameters.ipynb

Unit tests, for a total coverage of 71%, are performed in test/NeuralNetworkTest.py:

File Coverage
layer.py 100%
loss_functions.py 97%
neural_network.py 80%
neuron.py 84%
utils.py 14%
NeuralNetwokTest.py 96%

Here is the minimal amount of code to create a network and a dataset, and fit the network on the dataset:

from neural_network import *
import numpy as np

# training set
x_train = np.array([ [x] for x in np.linspace(0, 2, 50)])
y_train = np.sin(x_train) + x_train**2 + 2

# define network architecture
architecture = [1, 5, 1]
neurons = [InputNeuron, SigmoidNeuron, LinearNeuron]
net = Network(architecture, neurons)

# fitting
tr_loss, tr_error, _, _, _, gradient_norm = net.train_SGD(x_train, y_train,
                                                             x_test=None,
                                                             y_test=None,
                                                             lossObject=SquaredError(),
                                                             epochs=1000,
                                                             learning_rate=0.01,
                                                             batch_size=16,
                                                             momentum=0.9,
                                                             regularization=0.001,
                                                             epsilon=1e-2)

which prints stop: norm gradient. Epoch 140, meaning that the optimization has stopped after 140 iterations due to the satisfaction of the stopping criterion on the norm of the gradient.