Neural Networks have become one of the most successful universal machine learning algorithms.
They play a key role in enabling machine vision and speech recognition, and are increasingly
adopted in other application domains.
While the underlying computation is structurally simple, their computational complexity is enormous
and comes along with equally challenging memory requirements both in regards to capacity and access bandwidth.
This limits deployment in particular within energy constrained, embedded environments.
In order to address these implementation challenges, a broad spectrum of new customized and heterogeneous
hardware architectures have emerged, sometime referred to as deep learning processing units,
often accompanied with co-designed algorithms to extract maximum benefit out of the hardware.
Furthermore, numerous optimization techniques are being explored to reduce
compute and memory requirements while maintaining accuracy.
This results in an abundance of algorithmic and architectural choices, some of which fit specific use cases
better than others.
For system level designers, there is currently no good way to compare the variety of hardware, algorithm and
optimization options. While there are many benchmarking efforts in this field, they cover only subsections of
the embedded design space. None of the existing benchmarks support essential algorithmic optimizations such as
quantization, an important technique to stay on chip, or specialized heterogeneous hardware architectures.
We propose a novel benchmark suite, named QuTiBench, that addresses this need.
QuTiBench is a novel multi-tiered benchmarking methodology that supports algorithmic optimizations such as
quantization and helps system developers understand the benefits and limitations of these novel compute architectures
in regards to specific neural networks and will help drive future innovation.
We invite the community to contribute to QuTiBench in order to be able to support the full spectrum of choices
in implementing machine learning systems.
See the website for instructions on contributing.
Blott, Michaela, et al. "QuTiBench: Benchmarking neural networks on heterogeneous hardware." ACM Journal on Emerging Technologies in Computing Systems (JETC) 15.4 (2019): 1-38. https://arxiv.org/pdf/1909.05009.pdf
Michaela Blott, Johannes Kath, Lisa Halder, Yaman Umuroglu, Nicholas Fraser, Giulio Gambardella, Miriam Leeser, and Linda Doyle. 2020. "Evaluation of Optimized CNNs on FPGA and non-FPGA based Accelerators using a Novel Benchmarking Approach." In The 2020 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA ’20). Association for Computing Machinery, New York, NY, USA, 317. https://dl.acm.org/doi/10.1145/3373087.3375348 Poster available here: https://github.com/michaelablott/QuTiBench/blob/master/Publications/FPGA2020_EvalCNNs_Poster.pdf
Michaela Blott , Nicholas J. Fraser , Giulio Gambardella , Lisa Halder, Johannes Kath, Zachary Neveu, Yaman Umuroglu , Member, IEEE, Alina Vasilciuc, Miriam Leeser , Senior Member, IEEE, and Linda Doyle, "Evaluation of Optimized CNNs on Heterogeneous Accelerators using a Novel Benchmarking Approach" in IEEE Transactions on Computers, vol. , no. 01, pp. 1-1, 5555. https://doi.ieeecomputersociety.org/10.1109/TC.2020.3022318
All info on how to customize the site is located on the fastpages github. The platform is fairly new, so as changes are added to the faspages repo it's not a bad idea to update this website to reflect that. Updating instructions are available also through the fastpages github.