NNNumpy

NNNumpy is a simple artifitial neural networks library implemented using numpy (python 3.6). The code is not efficient (when using convolution layers), but it is very clear.

Now there is one model: sequential neural network.

Layers

FullyConnected (with one of activation functions)
Activation
Dropout
BatchNormalization
Flatten
Convolution2D
MaxPool2D

Activation functions (you can run activations.py file to see graphics)

Linear
TanH
ReLU
SoftPlus
Sigmoid
ArcTan
ISRU
BentIdentity
Sinusoid
Gaussian
SoftMax

Regularizations

L1Regularization
L2Regularization
L1L2Regularization

Losses

AbsoluteError (mean absolute error)
SquaredError (mean squared error)
CrossEntropy (categorial)

Optimizers

SGD
NesterovAG (Nesterov accelerated gradient)
Adagrad
RMSprop
Adam

Helpers

LRScheduler (learning rate scheduler)
EarlyStopper
ModelSaver

Examples

1. Regression (regression_test.py)

Model fitted on noised data (blue and orange dots) predictions

2. Classification (classification_test.py)

It is the dataset

And predictions after fitting (~97% accuracy on the whole dataset)

3. MNIST

You can use a pre-trained models (re-train it) or train your own models.

MLP (mnist_mlp_test.py)
This model after 15 epochs gets to 97.3% validation accuracy.

NeuralNetwork (7 layers) input_shape: (None, 784)
no. : Layer name (output_shape) : description
1 : FullyConnected (None, 400) : ArcTan(-Pi/2,Pi/2)(314000 params)
2 : BatchNormalization (None, 400) : (800 params)
3 : Dropout (None, 400) : None
4 : FullyConnected (None, 400) : ArcTan(-Pi/2,Pi/2)(160400 params)
5 : BatchNormalization (None, 400) : (800 params)
6 : Dropout (None, 400) : None
7 : FullyConnected (None, 10) : SoftMax(probabilities)(4010 params)
Total params num: 480010

CNN (mnist_cnn_test.py)
And this model after 5 epochs gets to 98.45% validation accuracy.

NeuralNetwork (9 layers) input_shape: (None, 1, 28, 28)
no. : Layer name (output_shape) : description
1 : Convolution2D (None, 12, 28, 28) : 12 3x3 ReLU[0,inf)(120 params)
2 : MaxPool2D (None, 12, 14, 14) : 2x2
3 : Convolution2D (None, 24, 14, 14) : 24 3x3 ReLU[0,inf)(2616 params)
4 : MaxPool2D (None, 24, 7, 7) : 2x2
5 : Convolution2D (None, 48, 7, 7) : 48 3x3 ReLU[0,inf)(10416 params)
6 : Flatten (None, 2352) : None
7 : BatchNormalization (None, 2352) : (4704 params)
8 : FullyConnected (None, 1176) : ReLU[0,inf)(2767128 params)
9 : FullyConnected (None, 10) : SoftMax(probabilities)(11770 params)
Total params num: 2796754

4. Cifar-10

There are also pre-trained models that you can check (re-train) or train your own models.

MLP
This model gets to 51.13% validation accuracy after 10 epochs.

NeuralNetwork (6 layers) input_shape: (None, 3072)
no. : Layer name (output_shape) : description
1 : FullyConnected (None, 350) : ArcTan(-Pi/2,Pi/2)(1075550 params)
2 : BatchNormalization (None, 350) : (700 params)
3 : FullyConnected (None, 300) : ArcTan(-Pi/2,Pi/2)(105300 params)
4 : BatchNormalization (None, 300) : (600 params)
5 : FullyConnected (None, 200) : ArcTan(-Pi/2,Pi/2)(60200 params)
6 : FullyConnected (None, 10) : SoftMax(probabilities)(2010 params)
Total params num: 1244360

CNN
And this model gets to 64.4% validation accuracy after 12 epochs.