/collage

System for automated integration of deep learning backends.

Primary LanguagePythonApache License 2.0Apache-2.0

Collage

System for automated integration of deep learning backends.

Installation

Since our implementation uses TVM as the main code generator, install tvm under tvm/.

  1. Install dependencies
sudo apt-get update
sudo apt-get install -y python3 python3-dev python3-setuptools gcc libtinfo-dev zlib1g-dev build-essential cmake libedit-dev libxml2-dev

pip3 install --user numpy decorator attrs tornado psutil xgboost cloudpickle pytest
  1. Install backends (e.g., cuDNN, MKL,...).
  • For cuDNN, install front-end library (commit 360d6e7164dfb7c802493fd1c0464f0d815b852a, tag: v0.1)
  1. Create build directory and go to build directory
mkdir tvm/build && cd tvm/build
  1. Prepare cmake configuration file. Make sure backend libaries of interest are built together. We provide cmake config that we used for our GPU/CPU experiments (config.cmake.gpu, config.cmake.cpu) in tvm/cmake/. Users may copy it to their build directory and rename it to config.cmake
cp ../cmake/config.cmake.gpu config.cmake
  1. Run cmake and make
cmake .. && make -j$(nproc)
  1. Declare following environment variables
export COLLAGE_HOME=/path/to/collage/repo
export COLLAGE_TVM_HOME=${COLLAGE_HOME}/tvm
export PYTHONPATH=${COLLAGE_TVM_HOME}/python:${COLLAGE_HOME}/python:${PYTHONPATH}

For more details, please reference TVM installation guide

  1. Install Collage dependencies
pip3 install --user graphviz bitarray deap pandas

If you face issues, try

pip3 install --upgrade protobuf
sudo apt-get install graphviz

Demo

Install the following dependencies for deep learning models used for demo.

pip3 install --user torch==1.7.1 torchvision==0.8.2 tqdm protobuf onnx onnxruntime opencv-python transformers

We provide two demos (demo_performance.py, demo_customization.py) under demo/.

  • demo_performance.py shows how collage optimizes given workloads with popular backends that Collage provides by default.
  • demo_customization.py shows how users can register new backend with their custom codegen, pattern, pattern rule.

For the best result, it is highly recommend to create the tuning log by using autotune_tvm_ops.py before running those demos.

Note

  • As Collage uses TVM as its code generator, it cannot support backends that TVM is unable to build. Tested backends are
    • TVM w/o tuning
    • TVM w/ AutoTVM
    • cuDNN v8.0.5
    • cuBLAS v11.3.0
    • TensorRT v7.2.2
    • MKL
    • DNNL
  • Since Collage is essentially a profile-guided search, variance in the measurement may affect the final backend placement. For the best result, multiple runs are highly recommended.
  • Due to some issues in the implementation, current evolutionary search only supports network implemented in get_network_from_torch(). If an user want to try new network, the network must be implemented within this function.

Cite

@article{jeon2021collage,
  title={Collage: Automated Integration of Deep Learning Backends},
  author={Jeon, Byungsoo and Park, Sunghyun and Liao, Peiyuan and Xu, Sheng and Chen, Tianqi and Jia, Zhihao},
  journal={arXiv preprint arXiv:2111.00655},
  year={2021}
}