Welcome to libfairydust!
libfairydust is a wrapper around NVIDIAs cuda and OpenCL implementation.
AMDs `stream' (APP) OpenCL implementation is also supported
Report bugs and problems to <adrian@blinkenlights.ch>
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
FAQ Index:
1.0 Why should i use libfairydust?
1.1 Great! How can i compile libfairydust?
2.0 How to use libfairydust
2.1 Playing with the enviroment (local testing)
2.2 Using fairyd (cluster installation)
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
1.0 Why should i use libfairydust?
--------------------------------------------------
libfairydust was written to be used in GPU-Clusters where multiple GPUs
are attached to one physical host and each GPU can be used by a different
user.
Most people do stuff like this:
...
nvret = cudaSetDevice(0)
if(nvret != CUDA_SUCCESS) {
die_in_some_fata_way("oh noes!");
}
...
This works great on your local workstation with one GPU but fails in
spectacular ways on a GPU cluster. In the best case your code will simply crash.
In the worst case *all* your processes will just run on GPU#0 and you won't
even know about it.
Libfairydust solves this problem by simulating `fake-devices': If an application
requests GPU#0, libfairydust can re-route all requests to some other GPU!
This requires *NO* changes to existing applications: The application won't even
know about this: libfairydust works 100% transparent.
1.1 Great! How can i compile libfairydust?
--------------------------------------------------
Libfairydust has been tested with Cuda 3.2 and gcc 4.1.2 + 4.4.4 on x86_64 linux. Support for 32bit installations is currently not implemented.
Compiling it is as easy as doing:
$ cd libfairydust/src
$ make
This should produce a file called ./obj/libfairydust.so.0
(Note: The makefile expects to find all cuda headers at
/usr/local/cuda/include OR/AND at $(CUDA_INSTALL_PATH)/include)
To compile an AMD-Compatible version run:
$ cd libfairydust/src
$ make atidust
This should produce a file called ./obj/libatidust.so.0
(Note: The rest of this documentation will talk about
`libfairydust.so.0' - the ati version works in exactly
the same way - just replace the library name when needed)
2.0 How to use libfairyudst
--------------------------------------------------
libfairydust.so.0 needs to get pre-loaded by ld-linux.
To do this, run something like this in your shell:
$ export LD_PRELOAD=/path/to/obj/libfairydust.so.0
That's it: Libfairydust will now hijack all cuda/openCL calls!
2.1 Playing with the environment (local testing)
--------------------------------------------------
Running any cuda/openCL application with libfairydust preloaded
should now show such an error message:
$ ./deviceQuery
./deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
fairydust.c starting up - (C) 2010-2011 Adrian Ulrich <adrian@blinkenlights.ch>
fairydust.c release git-108, compiled with gcc-4.4.4 and cuda-3020 at 1295351458
connection to fairyd failed: aborting
Booh! What happened? Libfairyd asked 'fairyd' for some new GPUs but fairyd isn't
running yet.
Don't worry: We can use libfairydust without a running fairyd instance:
$ export FDUST_FORCE_DEBUG=1 # get some debug infos
$ FDUST_ALLOCATE=0 ./deviceQuery
`deviceQuery' will now show exactly ONE GPU (even if you have 10 GPUs)
If you have multiple GPUs you can try to run
$ FDUST_ALLOCATE=1 ./deviceQuery
It will still show only one GPU, but this time libfairydust tricked libcuda into
beliving that GPU#1 is actually GPU#0
You might also have noticed that libfairydust changed the device names into something
like this.
Device = GeForce 9500 GT - fdust{v:h}={0:1}
In this case libfairydust simulated a VIRTUAL (v) device 0.
The REAL (h (hardware)) device for this virtual device is set to device 1
`FDUST_ALLOCATE' also supports multiple GPUs: Setting it to
"0 3 2"
Will map:
The REAL GPU0 to the VIRTUAL GPU0
The REAL GPU3 to the VIRTUAL GPU1
The REAL GPU2 to the VIRTUAL GPU2
Setting FDUST_ALLOCATE="@" will cause libfairydust to return all found GPUs
2.2 Using fairyd (cluster installation)
--------------------------------------------------
In a 'real-world' installation you will have to adapt fairyd.pl to your batchsystem.
The included version of fairyd.pl works with LSF 7.x and was written for use on the
Brutus-Cluster of the ETH Zurich.
Blabla: Fixme: Write more text :-)
- add startup workflow (connect to -> reply)
- protocol description
- testing implementation (use xterm as launcher?)