0. About
Shadowfax Distributed GPGPU Assembly Runtime
Author: Alexander Merritt, merritt.alex@gatech.edu (main author)
Contributors (direct or indirect):
Magda Slawinska, magg@gatech.edu (some marshaling/RPC code, old build system)
Abhishek Verma (MS - RPC protocol and paper in VTDC'11)
Vishakha Gupta (original marshaling code and virtualization work)
This implementation of the 'Shadowfax' software is based on GViM developed by
Vishakha Gupta (published in GPUVirt 2009) for virtualized systems, providing a
distributed runtime for applications to transparently utilize GPUs across a
cluster. Code originally in GViM is now scarce in Shadowfax (e.g. the
cuda_packet specification) and will continue to disappear.
1. Building the code
Dependencies: scons (python), NVIDIA CUDA, GNU GCC, LLVM Clang, gpu-remote
gpu-remote provides an interposer (and soon marshaling code) as a separate
project. Obtain here:
$ git clone https://bitbucket.org/alex_merritt/gpu-remote.git
$ cd src/CUDA/vX.Y/preload
$ scons
# scons install
# ldconfig
Build Shadowfax. This produces two binaries: src/shadowfax and
src/lib/libsfcuda.so
$ scons [debug=1] [network=sdp]
By default, Ethernet is selected. If CUDA is installed in another location
$ CUDA_ROOT=/path/to/cuda scons [debug=1] [network=sdp]
One instance of 'shadowfax' must be run on each node
node0 $ cd build/bin/; ./shadowfax main
node1 $ cd build/bin/; ./shadowfax minion ip-of-main
node2 $ cd build/bin/; ./shadowfax minion ip-of-main
[...]
To clean
$ scons -c
Keeneland - use the GNU Compiler toolchain (instead of ICC or PGC) and point the
appropriate cuda variables in the SConstruct file to the location of the NVIDIA
toolchain/libraries.
2. Applications
i. Launch shadowfax daemons (master before slaves)
ii. On each node where there is a process belonging to the application, create
an 'assembly hint' file describing to the shadowfax system what properties of
GPUs and assemblies are desired.
mpi=N
policy=[local_first|remote_only]
batch_size=M
N=0 -> not an MPI program
N>0 -> all hints with same N-value indicate process belongs to same program
M=0 -> use default batch size (8192 packets)
M>0 -> set batch to flush when full or on M RPC packets (M <= 8192)
local_first -> for each vGPU given to process
find first unmapped local GPU, else
share some local GPU
remote_only -> for each vGPU given to process
find first unmapped remote GPU, else
share some remote GPU, else
fallback to local_first policy
Run applications which were linked against NVIDIA's libraries using the
following method:
$ LD_LIBRARY_PATH=src/lib LD_PRELOAD=libsfcuda.so:libprecudart.so ./app
If you specify an assembly hint:
$ ASSEMBLY_HINT=$(pwd)/hintfile \
LD_LIBRARY_PATH=src/lib LD_PRELOAD=libsfcuda.so:libprecudart.so ./app
3. Issues
Compatibility for CUDA 4.x under development
Keeneland - If your CUDA root directory is not /usr/local/cuda you can specify
it as:
export CUDA_ROOT=$(cd path/to/root && pwd)