PyNet

PyNet is a easy and didactic framework for machine learning with neural networks. The focus of the framework is understanding in a easy way the basic concepts of a a neural network thanks to a short and clear implementation in python. In the framework are implemented some basic networks as Hopfield network, Kohonen network, Q-Learing network, Vanilla network, and it is easy to implement every layers networks.

The Architecture

The architecture of PyNet is pretty similar to Torch7 and Keras interfaces. The main modules of the framework are:

genericlayer.py. It is the main common classes.
layers.py. All the main classic layers already implemented as:
- LinearLayer: linear layer with bias
- MWeightLayer: linear layer without bias
- VWeightLayer: layer of weight
- SoftMaxLayer
- SigmoidLayer
- TanhLayer
- etc...

network.py. This file contains the main structure for create a computational graph as:

Sequential: each element in sequential has to have the number of inputs equal to the number of outputs of the previous layer. The input is forword across the chain in the sequential elements.
Parallel: each element in parallel has to have the same number of inputs, the inputs is forwarded to each element of in the parallel.

An Example:

model = Sequential(
    LinearLayer(2, 5),
    SigmoidLayer,
    LinearLayer(5, 3),
    SoftMaxLayer
)
#is the same of
model = Sequential(
    LinearLayer(2, 5),
    SigmoidLayer,
    Parallel(
        LinearLayer(5, 1),
        LinearLayer(5, 1),
        LinearLayer(5, 1),
    )
    SoftMaxLayer
)

These two classes manage a vector input and generate a vector output. Moreover there are some structures to manage group of vector, in order to create a computational graph as:

SumGroup
MulGroup
ParallelGroup
MapGroup

An Example:

#y = a*x^2+b*x+c
n = Sequential(
        ParallelGroup(
            Sequential(
                ParallelGroup(
                    GenericLayer,  # x
                    GenericLayer,  # x
                    VWeightLayer(1) # a
                ),MulLayer # x*x*a
            ),
            Sequential(
                ParallelGroup(
                    GenericLayer,   # x
                    VWeightLayer(1), # b
                ),MulLayer # b*x
            ),
            VWeightLayer(1) # constant c
        ),SumLayer # sum of all terms
    )
#is the same of
n2 = Sequential(
        ParallelGroup(
            Sequential(
                ParallelGroup(
                    GenericLayer,  # x
                    GenericLayer,  # x
                    VWeightLayer(1) # a
                ),MulLayer # x*x*a
            ),
            LinearLayer(1,1) # b*x+c
        ),SumLayer # sum of all terms
    )
#is the same of
n = Sequential(
        ParallelGroup(GenericLayer,GenericLayer,GenericLayer), # create a vector of [x,x,x]
        SumGroup( # the vector is used in:
            Sequential( # x
                ParallelGroup(GenericLayer,GenericLayer,GenericLayer), # create a vector of [x,x,x]
                MulGroup(
                    GenericLayer, # x
                    GenericLayer, # x
                    VWeightLayer(1) # a
                ) # x^2*a
            ),
            Sequential( # x
                ParallelGroup(GenericLayer,GenericLayer), # create a vector [x,x]
                MulGroup(
                    GenericLayer, # x
                    VWeightLayer(1) # b
                ) # x*b
            ),
            VWeightLayer(1) # c
        )
    )

computationalgraph.py. This file contains a group of classes for a fast creation of complex graph as:
- Input: class that take a list string as variables used in the computational graph and the chosen varible
- VWeight: class to create a vector of weight
- MWeight: class to create a matrix of weight
- Sigmoid: class to create a sigmoid layer
- Tanh: class to create a Tanh layer
The computationalgraphlayer is a layer as all the other so it implements the function forward and backward.

Referring to the case shown before of "y = ax^2+bx+c" the code become:
```
#y = a*x^2+b*x+c
x = Input(['x'],'x')
a = VWeight(1)
b = VWeight(1)
c = VWeight(1)
n = ComputationalGraphLayer(a*x**2+b*x+c)
```
An example of classic linear layer with a sigmoid:
```
#y = Sigmoid(W*x+b)
x = Input(['x'],'x')
W = MWeight(5,3) #5 input 3 output
b = VWeight(3)
n = ComputationalGraphLayer(Sigmoid(W*x+b))
```
losses.py. In this file there are the losses function as:
- SquareLoss
- NegativeLogLikelihoodLoss
- CrossEntropyLoss
optimizer.py. Here there are the main optimizer classes as:
- GradientDescent
- GradientDescentMomentum
trainer.py. In this file there is the class for the training.

Complete Example of classification

    train = #GET TRAIN DATA

    from layers import LinearLayer, SoftMaxLayer, SigmoidLayer
    from losses import NegativeLogLikelihoodLoss
    from optimizers import GradientDescentMomentum
    from network import Sequential
    from trainer import Trainer
    from printers import Printer2D, ShowTraining

    num_classes = 4
    classes = ["o","^","x","s"]
    colors = ['r', 'g', 'b', 'y']

    model = Sequential(
        LinearLayer(2, 3),
        SigmoidLayer,
        LinearLayer(3, num_classes),
        SoftMaxLayer,
    )

    display = ShowTraining(epochs_num = 5)
    trainer = Trainer(show_training = True, show_function = display.show)

    J_train_list, dJdy_list = trainer.learn(
        model = model,
        train = train,
        loss = NegativeLogLikelihoodLoss(),
        optimizer = GradientDescentMomentum(learning_rate=0.1, momentum=0.1),
        batch_size = 10,
        epochs = 5
    )

    for i, (x,target) in enumerate(train):
        y.append(model.forward(x))

    p = Printer2D() # A class just for the 2D input of classification
    p.draw_decision_surface(1, model, train)
    p.compare_data(1, train, y, num_classes, colors, classes)
    plt.title('After Training')

Why

The only good way to understand modern neural network is to try to implement one from scratch. After that you can use TensorFlow.

tonegas/PyNet

PyNet

The Architecture

Complete Example of classification

Why