neon

neon is Nervana's Python based Deep Learning framework. We have designed it with the following functionality in mind:

YAML for easy model specification (inspired by pylearn2)
Python for easily adding models and support for many data formats
Support for commonly used models: convnets, MLPs, RNNs, LSTMs, autoencoders, RBMs
Support for common learning rules, activation functions and cost functions
Comparing performance of alternate numeric representations with 32-bit floating point (fp32) for Deep Learning
Support for using spearmint for hyperparameter optimization
Swappable hardware backends: write code once and then deploy on CPUs, GPUs, or Nervana hardware

Features that are unique to neon include:

Tight integration with nervanagpu kernels for fp16 and fp32 (benchmarks) on Maxwell GPUs. These are the fastest implementations of the benchmark deep networks.
3s/macrobatch on AlexNet on Titan X (Full run on 1 GPU ~ 32 hrs)
Out of the box fp16 AlexNet model that has the same accuracy as fp32
Integration with our fork (cudanet) of Alex Krizhevsky's cuda-convnet2 library for Kepler GPU support (note that the v0.9.0 neon release breaks support for cudanet under certain types of data/models)
Support for our distributed processor (Nervana Engine™) for deep learning.

We use neon internally at Nervana to solve our customers' problems across many domains. We are hiring across several roles. Apply here!

Getting started

Basic information to get started is below. Please consult the full documentation for more information.

Installation

Local install and dependencies
Cloud-based access (email us for an account)
Docker images (community provided)

Quick Install

On a Mac OSX or Linux machine, enter the following to download and install neon, and use it to train your first multi-layer perceptron or convolutional neural networks below.

git clone https://github.com/NervanaSystems/neon.git
cd neon
sudo make install

The above will install neon system-wide. If you don't have sufficient privileges or would prefer an isolated installation, see either our virtualenv based install, or take a look at some of the community provided docker images.

There are several examples built-in to neon in the examples directory for a user to get started. The YAML format is plain-text and can be edited to change various aspects of the model. See the ANNOTATED_EXAMPLE.yaml for some of the definitions and possible choices.

Running a simple MNIST model (on CPU)

neon examples/mlp/mnist-small.yaml

Running an Alexnet model (on Maxwell GPU)

In fp32:

# for nervangpu (requires Maxwell GPUs)
neon --gpu nervanagpu examples/convnet/i1k-alexnet-fp32.yaml

In fp16:

neon --gpu nervanagpu examples/convnet/i1k-alexnet-fp16.yaml

Distributed across 4 Maxwell GPUs in the same machine ("weird-trick" style parallelization):

neon --gpu nervanagpu4 examples/convnet/i1k-alexnet-fp32.yaml

Code organization

backends    --- implementation of different hardware backends
datasets    --- support for common datasets CIFAR-10, ImageNet, MNIST etc.
diagnostics --- hooks to measure timing and numeric ranges
hyperopt    --- hooks for hyperparameter optimization
layers      --- layer code
models      --- model code
optimizers  --- learning rules
transforms  --- activation & cost functions
metrics     --- performance evaluation metrics

Documentation

The complete documentation for neon is available here. Some useful starting points are:

Issues

For any bugs or feature requests please:

Search the open and closed issues list to see if we're already working on what you have uncovered.
Check that your issue/request has already been addressed in our Frequently Asked Questions (FAQ)
File a new issue or submit a new pull request if you have some code you'd like to contribute

Machine learning OPerations (MOP) Layer

The MOP is an abstraction layer for Nervana's system software and hardware which includes the Nervana Engine, a custom distributed processor for deep learning.

The MOP consists of linear algebra and other operations required by deep learning. Some MOP operations are currently exposed in neon, while others, such as distributed primitives, will be exposed in later versions as well as in other forthcoming Nervana libraries.

Defining models in a MOP-compliant manner guarantees they will run on all provided backends. It also provides a way for existing Deep Learning frameworks such as theano, torch, and caffe to interface with the Nervana Engine.

Upcoming libraries

We have separate, upcoming efforts on the following fronts:

Automatic differentiation
Major refactoring enhancements to make the interface cleaner and easier to use
Integration with Nervana Cloud™

License

We are releasing neon and nervanagpu under an open source Apache 2.0 License. We welcome you to contact us with your use cases.

lihuanghai/neon