Benchmarking Chainer with Airspeed Velocity.
You need to install asv benchmark framework and Cython (to build CuPy).
pip install asv CythonThe benchmark can be run over Chainer v4.0.0b1+.
To run one-shot benchmark against installed Chainer / CuPy:
asv run --python=same --show-stderrTo run benchmark to compare results between commits:
# Enable ccache for performance (optional).
export PATH="/usr/lib/ccache:${PATH}"
export NVCC="ccache nvcc"
# Run benchmark against target commit-ish of Chainer and CuPy.
# Note that specified versions must be a compatible combination.
# You can use `find_cupy_version.py` helper tool to get appropriate CuPy
# version/commit for the given Chainer version/commit.
./run.sh master master
./run.sh v4.0.0b4 v4.0.0b4
# Compare the benchmark results between two commits to see regression
# and/or performance improvements in command line.
alias git_commit='git show --format="%H"'
asv compare $(git_commit v4.0.0b4) $(git_commit master)
# Convert the results into HTML.
# The result will be in `html` directory.
asv publish
# Start the HTTP server to browse HTML.
asv previewAlternatively you can use Docker.
# Build docker image for benchmark.
docker build -t chainer-benchmark docker
# Create a machine configuration file (`.asv-machine.json`) in this directory (first time only).
nvidia-docker run --rm -u $(id -u):$(id -g) -v ${PWD}:/benchmarks -w /benchmarks -e HOME=/benchmarks chainer-benchmark asv machine --machine $(hostname)
# Run benchmark.
nvidia-docker run --rm -u $(id -u):$(id -g) -v ${PWD}:/benchmarks -w /benchmarks -e HOME=/benchmarks chainer-benchmark ./run.sh master master --machine $(hostname)