numpy implementation of Recurrent Neural Networks
- RNN_numpy based on iamtrask's github.io
- LSTM_numpy based on wiseodd's github.io
Training Vanilla RNN for binary addition.
largest = pow(2, BIN_DIM)
decimal = np.array([range(largest)]).astype(np.uint8).T
binary = np.unpackbits(decimal, axis=1)Simply prepare the binary digit database. And propagate forward and backward to train binary addition.
hidden = sigmoid(np.dot(X, w0) + np.dot(hidden_values[-1], wh))
output = sigmoid(np.dot(hidden, w1))
pred[pos] = np.round(output[0][0])Training was successful with 100% accuracy.
----------
Iter 9000
Error : [ 0.68126419]
Pred : [1 0 1 0 1 0 0 0]
True : [1 0 1 0 1 0 0 0]
108 + 60 = 168
----------
Training LSTM RNN for mnist handwritten digit recognition.
mnist = tf.keras.datasets.mnist
(train_x, train_y), _ = mnist.load_data()Prepare the mnist digit database with tf.keras.datasets.mnist.
caches, states = LSTM_Cell(Xt)
c, h = states[-1]
out = np.dot(h, wy) + by
pred = softmax(out)
entropy = cross_entropy(pred, Y)Train the LSTM model with BPTT and it was successful to recognize handwritten digits.
