A list of benchmark suites (and some loose kernels) used in the research related to compilers, program performance, scientific computations etc.
| Benchmark | Lang | Platform | Published in | Content | Additional info |
|---|---|---|---|---|---|
| BEEBS | C | Embedded | [Pallister2013] | 10 benchmarks | For analysis of energy consumption on embedded platforms |
| Embench™ | C | Embedded | [Patterson2019] | 19 benchmarks | Modern benchmarks for performance analysis built on top of selected BEEBS benchmarks |
| GridNBP 3.1 | Fortran, Java (v3.0) | CPU | [VanDerWijngaart2002] | 4 benchmarks | Evaluation of the computational grids, their communications and distributed tasks. |
| LCALS v1.0.2 | C | CPU | [Hornung2013] | 32 loops | For analysis of compiler optimisations; modern extension to Livermore loops; more information |
| Livermore loops (LFK) | FORTRAN | CPU | [McMahon1986] | 24 loops | For performance analysis |
| Livermore loops | C | CPU | 1992 | 24 loops | Port to C |
| Mälardalen WCET Benchmarks | C | CPU | [Gustafsson2010] | 35 benchmarks | For Worst-Case Execution Time (WCET) analysis |
| MiBench | C | Embedded | [Guthaus2001] | 6 benchmarks | For performance analysis |
| NAS Parallel Benchmarks 3.4.2 | C, Fortran, MPI, OpenMP, Java (v3.0) | CPU | [Bailey1991] | 5 kernels, 3 programs, 4 other benchmarks | Evaluation of the performance of parallel supercomputers; more information |
| PARKBENCH 2.1.1 | Fortran, MPI, PVM | CPU | [Hockney1994] | 10 measurement codes, 7 kernels, 3 application | For performance analysis of parallel architectures; more information |
| PARSEC 3.0 | C, OpenMP, pthreads | CPU | [Bienia2008] | 13 programs | For research on parallelization; other versions |
| PolyBench/ACC | C, CUDA, OpenCL, OpenACC, HMPP, OpenMP | GPU, accelerators | [Grauer-Gray2012] | 29 kernels | Updated PolyBench/GPU; GitHub |
| PolyBench/C 4.2 | C | CPU | 2010 | 30 kernels | For the analysis of performance and compiler optimisations (especially related to polyhedra compilation); more information; SourceForge |
| PolyBench/Fortran 1.0 | FORTRAN | CPU | 2011 | 30 kernels | Port to FORTRAN; more information |
| PolyBench/GPU 1.0 | C, CUDA, OpenCL, OpenHMPP | CPU, GPU | [Grauer-Gray2012] | 15 kernels | Port to heterogeneous architectures; more information |
| SPLASH | C | CPU | [Singh1992] | 4 kernels, 8 programs | For research on parallelization; source code unavailable |
| SPLASH-2 | C | CPU | [Woo1995] | 4 kernels, 8 programs | For research on parallelization; original source code is unavailable, several modifications: staceyson/splash2, liuyix/splash2_benchmark, andysan/splash2 |
| SPLASH-3 | C | CPU | [Sakalis2016] | 4 kernels, 8 programs | For research on parallelization |
| TSVC | FORTRAN | CPU | [Callahan1988] | 135 loops | For testing automatic vectorizing compilers; Single precision version |
| TSVC | C | CPU | [Maleki2011] | 151 loops | Extended port to C |
| TSVC 2 | C | CPU | 2015 | 151 loops | Update to TSVC in C |
| UTDSP | C | CPU | 1992 | 6 loops, 12 programs | For testing compilers on Digitial Signal Processing (DSP) applications; more information |
| VersaBench | C | Custom CPUs | [Rabbah2004] | 15 programs | For the performance analysis of new flexible CPU architectures |
- BEEBS — Bristol/Embecosm Embedded Benchmark Suite
- LCALS — Livermore Compiler Analysis Loop Suite
- LFK — Livermore FORTRAN Kernels
- NPB — NAS Parallel Benchmarks
- PARKBENCH — PARallel Kernels and BENCHmarks
- PARSEC — The Princeton Application Repository for Shared-Memory Computers
- PolyBench — The Polyhedral Benchmark Suite
- SPLASH — Stanford Parallel Applications for Shared-Memory
- TSVC — Test Suite for Vectorizing Compilers
- VersaBench — A Benchmark Suite for Versatile Architectures
- [Bailey1991] D. H. Bailey et al., "The NAS parallel benchmarks summary and preliminary results," Supercomputing '91: Proceedings of the 1991 ACM/IEEE Conference on Supercomputing, 1991, pp. 158-165, doi: https://doi.org/10.1145/125826.125925.
- [Bienia2008] Christian Bienia, Sanjeev Kumar, Jaswinder Pal Singh, and Kai Li. 2008. "The PARSEC benchmark suite: characterization and architectural implications". In Proceedings of the 17th international conference on Parallel architectures and compilation techniques (PACT '08). Association for Computing Machinery, New York, NY, USA, 72–81. doi:https://doi.org/10.1145/1454115.1454128
- [Callahan1988] Callahan, D., Dongarra, J., & Levine, D. (1988). "Vectorizing compilers: a test suite and results". In Proceedings. SUPERCOMPUTING ’88 (pp. 98–105). IEEE Comput. Soc. Press. https://doi.org/10.1109/SUPERC.1988.44642
- [Guthaus2001] M. R. Guthaus, J. S. Ringenberg, D. Ernst, T. M. Austin, T. Mudge and R. B. Brown, "MiBench: A free, commercially representative embedded benchmark suite," Proceedings of the Fourth Annual IEEE International Workshop on Workload Characterization. WWC-4 (Cat. No.01EX538), Austin, TX, USA, 2001, pp. 3-14, doi: https://doi.org/10.1109/WWC.2001.990739
- [Grauer-Gray2012] Scott Grauer-Gray, Lifan Xu, Robert Searles, Sudhee Ayalasomayajula, and John Cavazos. "Auto-tuning a High-Level Language Targeted to GPU Codes". Proceedings of Innovative Parallel Computing (InPar '12), 2012. https://doi.org/10.1109/InPar.2012.6339595
- [Gustafsson2010] Jan Gustafsson, Adam Betts, Andreas Ermedahl, and Björn Lisper. "The Mälardalen WCET benchmarks: Past, present and future". 10th International Workshop on Worst-Case Execution Time Analysis, WCET 2010, July 6, 2010, Brussels, Belgium. https://doi.org/10.4230/OASIcs.WCET.2010.136
- [Hockney1994] Roger Hockney, and Michael Berry. "Public International Benchmarks for Parallel Computers". Technical report. February 7, 1994. https://netlib.org/parkbench/parkbench.ps
- [Hornung2013] Richard D. Hornung and Jeffrey A. Keasler, “A Case for Improved C++ Compiler Support to Enable Performance Portability in Large Physics Simulation Codes”, LLNL-TR-635681 (2013). https://computing.llnl.gov/projects/co-design/compilersupportwhitepaper_tr-635681_cover.pdf
- [Maleki2011] Maleki, S., Gao, Y., Garzarán, M. J., Wong, T., & Padua, D. A. (2011). "An evaluation of vectorizing compilers". Parallel Architectures and Compilation Techniques - Conference Proceedings, PACT, 7, 372–382. https://doi.org/10.1109/PACT.2011.68
- [McMahon1986] F. H. McMahon. "Livermore fortran kernels: A computer test of numerical performance range". Technical Report UCRL-53745, Lawrence Livermore National Laboratory, Livermore, CA, December 1986; COMMENT: no PDF on the web.
- [Patterson2019] David Patterson, Jeremy Bennett, Palmer Dabbelt, Cesare Garlati, G. S. Madhusudan and Trevor Mudge. "Embench™: An Evolving Benchmark Suite for Embedded IoT Computers from an Academic-Industrial Cooperative: Towards the Long Overdue and Deserved Demise of Dhrystone" (2019). https://riscv.org/wp-content/uploads/2019/06/9.25-Embench-RISC-V-Workshop-Patterson-v3.pdf
- [Pallister2013] James Pallister, Simon Hollis, Jeremy Bennett (2013). "BEEBS: Open Benchmarks for Energy Measurements on Embedded Platforms" http://arxiv.org/abs/1308.5174.
- [Rabbah2004] Rodric M. Rabbah, Ian Bratt, Krste Asanovic, and Anant Agarwal, "Versatility and VersaBench: A New Metric and a Benchmark Suite for Flexible Architectures", MIT CSAIL Technical Memo, MIT-LCS-TM-646, June 2004, http://catfish.csail.mit.edu/~rabbah/versabench/MIT-LCS-TM-646.pdf
- [Sakalis2016] C. Sakalis, C. Leonardsson, S. Kaxiras and A. Ros, "Splash-3: A properly synchronized benchmark suite for contemporary research", 2016 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2016, pp. 101-111, doi: https://www.doi.org/10.1109/ISPASS.2016.7482078.
- [Singh1992] Jaswinder Pal Singh, Wolf-Dietrich Weber, and Anoop Gupta. 1992. "SPLASH: Stanford parallel applications for shared-memory". SIGARCH Comput. Archit. News 20, 1 (March 1992), 5–44. DOI:https://doi.org/10.1145/130823.130824
- [VanDerWijngaart2002] Rob F. Van der Wijngaart et al. "NAS Grid Benchmarks Version 1.0". NASA Technical Report NAS-02-005. July, 2002. https://www.nas.nasa.gov/assets/pdf/techreports/2002/nas-02-005.pdf
- [Woo1995] S. C. Woo, M. Ohara, E. Torrie, J. P. Singh and A. Gupta, "The SPLASH-2 programs: characterization and methodological considerations," Proceedings 22nd Annual International Symposium on Computer Architecture, 1995, pp. 24-36, doi: https://www.doi.org/10.1109/ISCA.1995.524546.