ONNX Runtime - Group20

This repository contains the group project for an ONNX Runtime implementation using Rust. The project is part of the Italian “System and device programming” course (02GRSOV) at Polytechnic University of Turin.

The requirements for the project are collected in issue #1.

Nearly all the development process is documented in the issues and in the pull requests. You can start from the requirements and then follow the development process to understand how the project was developed.

Project structure

It is organized using a cargo workspace with the following structure:

onnx-runtime: the main library that implements our ONNX Runtime
onnx-runtime-cli: a command line interface app that uses the library
onnx-binding: Python bindings for the library
onnx-runtime-pyapp: a Python app that uses our Python bindings

Code documentation (cargo doc)

If you want to generate the documentation for the project, you can run the following command:

cargo doc --no-deps

Modern browsers won't allow you to open the documentation if it is not served through a web server. If you have python installed, you can use the following command to quickly serve the documentation:

python -m http.server --directory target/doc/

Then you can open our library documentation at the following address: http://localhost:8000/onnx_runtime/

Testing (cargo test)

If you want to run all the tests for the project, you can run the following command:

cargo test --release

NOTE: the tests are run in release mode to speed up the execution. Otherwise, the tests would take too much time to complete.

Benchmarking

We use criterion to run benchmarks for our project.

If you want to run all the benchmarks, you can run the following command:

cargo bench

They can take a while to complete.

In alternative you can run a single benchmark with the following command:

cargo bench --bench <bench_name>

For example, this will run the benchmark for the mobilenet model with 4 threads and a cat image as input:

cargo bench --bench "Cat mobilenet 4"

The interesting benchmarks are the ones that have the following names:

Cat mobilenet: it runs the mobilenet model with a cat image as input in single thread mode
Cat mobilenet 2: it runs the mobilenet model with a cat image as input in multi thread mode (2 threads)
Cat mobilenet 4: it runs the mobilenet model with a cat image as input in multi thread mode (4 threads)
Cat resnet: it runs the resnet model with a cat image as input in single thread mode
Cat resnet 2: it runs the resnet model with a cat image as input in multi thread mode (2 threads)
Cat resnet 4: it runs the resnet model with a cat image as input in multi thread mode (4 threads)

Benchmark reports

The nice thing about criterion is that it generates a report with the results of the benchmarks. You can find the report in the target/criterion/report/index.html file.

Again, modern browsers won't allow you to open the HTML report if it is not served through a web server. If you have python installed, and want to use http.server module this time as well, you may have some problems because of the relative paths used in the report. To avoid that, you can host the criterion directory as a root folder and navigate to the report folder:

python -m http.server --directory target/criterion/

Then you can open the report at the following address: http://localhost:8000/report/index.html

Report example

This is an output example obtained using a machine with a i3-10100F CPU:

Resnet [4 threads]	Mobilenet [4 threads]