To maximize the resource efficiency of inference servers, we proposed a key mechanism to exploit hardware support for spatial partitioning of GPU resources. With the partitioning mechanism, a new abstraction layer of GPU resources is created with configurable GPU resources. The scheduler assigns requests to virtual GPUs, called Gpu-lets, with the most effective amount of resources. The prototype framework auto-scales the required number of GPUs for a given workloads, minimizing the cost for cloud-based inference servers. The prototype framework also deploys a remedy for potential interference effects when two ML tasks are running concurrently in a GPU.
- Ubuntu 18.04
- Linux Kernel 4.15
- CUDA 10.2
- cuDNN 7.6
- PyTorch 1.10
The prototype was evaluated with multi-GPU server with the following hardware:
- RTX 2080ti (11GB global memory)
- intel Xeon E5-2630 v4
- Servers connected with 10 GHz Ethernet
Install the following libraries and drivers to build the prototype
-
LibTorch(PyTorch library for C++) = 1.10
-
CUDA >= 10.2
-
CUDNN >= 7.6
-
Boost >= 1.6 (script provided)
-
opencv >= 4.0 (script provided)
-
cmake >= 3.19, use cmake to build binaries (script provided)
-
Download libtorch and extract all the content as 'libtorch' under the root directory of this repo. (example: 'glet/libtorch')
-
Go to 'scripts/'
cd scripts
- Execute 'build_all.sh'
./build_all.sh
The script will use cmake to auto-configure build environments and build binaries.
More Example scripts will be added in the future
Below are step-by-step examples for running the server and standalone components
Use 'execLocal.sh' to execute ML inference on local GPU. Useful for testing whether you have installed compatible SW stack and profiling latency.
-
Make sure you have downloaded models you want to execute and store them under 'resource/models/'.
-
Go to 'scripts/'
cd scripts
- Execute execLocal.sh with parameters: 1) name of model you want to execute, 2) number of executions 3) batch size 4) interval between executions 5) (optional) percentage of computing resource.
example) ./execLocal.sh resnet50 1000 1 0.1 50
Highly recommended that you replace example profile files with profile info on the platform you wish to execute
Below is a list of items/features that are planned to be added to this repo.
-
Source code of all SW components used in experimentation: standalone inference binary, standalone scheduler, request generator, backend server, frontend server and proxy server (completed)
-
Scripts used for executing and analyzing experiments (ongoing)
-
Docker related files e.g.) Dockerfile
-
Improved UI for SW components
-
Misc: Refactoring for better code, consistency
If you have any suggestions or questions feel free to send me an Email:
[ATC] "Serving Heterogeneous Machine Learning Models on Multi-GPU Servers with Spatio-Temporal Sharing", accepted for The 2022 USENIX Annual Technical Conference, July, 2022