Caffe-MPI for Deep Learning
Introduction
Caffe-MPI is developed by AI & HPC Application R&D team of Inspur. It is a parallel version for multi-node GPU cluster, which is designed based on the NVIDIA/Caffe forked from the BVLC/caffe ( https://github.com/NVIDIA/caffe, more details please visit http://caffe.berkeleyvision.org).
Features
(1) The design basics
The Caffe-MPI is designed for high density GPU clusters; The new version supports InfiniBand (IB) high speed network connection and shared storage system that can be equipped by distributed file system, like NFS and GlusterFS. The training dataset is read in parallel for each MPI process. The hierarchical communication mechanisms were developed to minimize the bandwidth requirements between computing nodes.
Updates of the Caffe-MPI 2.0
Support NCCL 2.0
Both inter and intra node GPU communication are managed by NCCL with GPU direct RDMA.
(2) High performance and high scalability
The AlexNet, GoogLeNet and ResNet model have been tested with Caffe-MPI 2.0 on a GPU cluster, which includes 4 nodes, and each of which has 4 P40 GPUs. The dataset is ImageNet. The speedup is 14.65X, 14.25X, 15.34X, for AlexNet (batchsize=1024), GoogLeNet (batchsize=128) and ResNet (batchsize=32) respectively on 4 nodes with 16 GPUs.
(3) Good inheritance and easy-using
Caffe-MPI retains all the features of the original Caffe architecture, namely the pure C++/CUDA architecture, support of the command line, Python interfaces, and various programming methods. As a result, the cluster version of the Caffe framework is user-friendly, fast, modularized and open, and gives users the optimal application experience.
Try your first MPI Caffe
This program can run 1 processes at least.
cifar10
- Run data/cifar10/get_cifar10.sh to get cifar10 data.
- Run examples/cifar10/create_cifar10.sh to conver raw data to leveldb format.
- Run mpi_train_quick.sh to train the net.
- Example of mpi_train_quick.sh script. mpirun -host node1,node2 -mca btl_openib_want_cuda_gdr 1 --mca io ompio -np 2 -npernode 1 ./build/tools/caffe train --solver=examples/cifar10/cifar10_quick_solver.prototxt --gpu=0,1,2,3
Reference
More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server
Deep Image: Scaling up Image Recognition
Ask Questions
For reporting bugs, please use the caffe-mpi/issues page or send email to us.
Email address: wush@inspur.com
Author
Shaohua Wu.