Mini-Keras
Keras like implementation of Deep Learning architectures from scratch using numpy.
How to contribute?
The project contains implementations for various activation functions, layers, loss functions, model structures and optimizers in files
activation.py, layer.py, loss.py, model.py and optimizer.py respectively.
Given below is list of available implementations (which may or may not require any improvements).
| Activation Functions | Status |
|---|---|
| Sigmoid | Available |
| ReLU | Required |
| Softmax | Required |
| Layer | Status |
|---|---|
| Dense | Available |
| Conv2D | Available |
| MaxPool2D | Available |
| Flatten | Available |
| BasicRNN | Required |
| Loss Function | Status |
|---|---|
| BinaryCrossEntropy | Available |
| CategoricalCrossEntropy | Required |
| Model Structure | Status |
|---|---|
| Sequential | Available |
| Optimizer | Status |
|---|---|
| GradientDescentOptimizer | Available |
| AdamOptimizer | Required |
| AdaGradOptimizer | Required |
| GradientDescentOptimizer (with Nesterov) | Required |
Each of the implementations are class-based and follows a keras like structure. A typical model training with Mini-Keras looks like this,
from model import Sequential
from layer import Dense, Conv2D, MaxPool2D, Flatten
from loss import BinaryCrossEntropy
from activation import Sigmoid
from optimizer import GradientDescentOptimizer
model = Sequential()
model.add(Conv2D, ksize=3, stride=1, activation=Sigmoid(), input_size=(8,8,1), filters=1, padding=0)
model.add(MaxPool2D, ksize=2, stride=1, padding=0)
model.add(Conv2D, ksize=2, stride=1, activation=Sigmoid(), filters=1, padding=0)
model.add(Flatten)
model.add(Dense, units=1, activation=Sigmoid())
model.summary()
model.compile(BinaryCrossEntropy())
print("Initial Loss", model.evaluate(X, y)[0])
model.fit(X, y, n_epochs=100, batch_size=300, learning_rate=0.003, optimizer=GradientDescentOptimizer(), verbose=1)
print("Final Loss", model.evaluate(X, y)[0])As you might have noticed, its very similar to how one will do it in Keras.
Testing new functionalities
The run.py consists of a small code snippet that can be used to test if your new implementation is working properly or not.
Implementation Details
All the implementations have a forward propagation and a backward propagation equivalent available as a method in the corresponding class. Below are the details for implementing all the functionalities under different categories.
README.ipynb explains each of the implementations with mathematical proofs for better understanding.