The compile-time feedforward neural network library that no one asked for, written in Rust.
It's a feedforward neural network
implementation (currently with only a single hidden layer), roughly following the methods described
in Michael Nielsen's Neural Networks and Deep Learning.
What's special about it is that the whole model, including the size of each layer, is defined at
compile time; this is possible in Rust using the compile-time numbers from the
typenum crate, and the nalgebra linear
algebra quite which leverages typenum.
If you want to define a neurotic neural network with 784 input neurons, 30 hidden neurons,
and 10 output neurons, initialized using a standard normal distribution, using a sigmoid activation
function and a mean-squared cost function, and optimized via stochastic gradient descent, it would
look like this:
let mut network: NeuralNetwork<U784, U30, U10, Sigmoid, MeanSquared> =
NeuralNetwork::new::<InputNormalizedNormal>();
let optimizer = StochasticGradientDescent::new(3., 30);Then, to train it with some data:
for i in 0..10 {
optimizer.optimize(&mut network, &training_images[..], &training_labels[..]);
}And finally, to get the output of the trained model for an input:
let y = network.feedforward(&test_images[0]);Since the sizes of all matrices involved in the algorithm are known at compile time, they can be stored in contiguous memory. In theory, this ought to be noticeably more efficient than any other CPU-based neural network implementation, since it makes much better use of CPU caches.
The only downside is a hideous and inflexible type-parameter-filled API!
It has the potential to be, maybe? I profiled it just enough to determine that the the matrix
multiplications in the backpropagation and feedforward steps were the vast majority of the CPU
load, but nothing beyond that. It's not even multitheaded; I tried, but nalgebra::Matrix currently doesn't
implement Send, which is necessary for parallel threads to pass data around.
Furthermore, this is a CPU-based implementation, so it's not going to compete with actual ML
frameworks which support GPU acceleration. This was a hobby project to explore neural networks and
to see if Rust's type was up to the task of building everything up at compile time. (For the
curious, the answer is: yes... with great pain. Take a look at the nalgebra Allocator usage
littered throughout the project.)
There is none, sorry; this isn't intended to be a proper library one might, you know, use. The code itself is quite well-documented though, so you could easily generate some prettier documentation from the source if you'd prefer that.
For a full working example, look at examples/digits.rs, which uses the MNIST handwritten digits
dataset to learn digit recognition. Run it with cargo run --release --example digits; after
1 iteration it will reach ~93% accuracy and by the end of the 10 iterations it ought to converge
to 95% or so. On my i5-4670K CPU this takes about 60 seconds.
This library has a limited number of implemented parameters, but it could easily be extended. Here's what's currently supported:
- Activation functions:
- Exponential
- Linear
- Sigmoid
- Cost functions:
- Mean Squared
- Weight Initializers:
- Standard Normal Distribution
- Input-normalized Normal Distribution
- Optimizers:
- Gradient Descent
- Stochastic Gradient Descent
One thing I would like to have implemented is support for deep networks; by using generic-array,
it would be totally doable to have a compile-time sized array of H*H weight matrices (where
H is the number of neurons in each hidden layer). I never got around to it.
This project is licensed under the WTFPL; you may use it for whatever purposes you'd like.