R7RS Benchmarks
Benchmarks for various Scheme implementations. Taken with kind permission from the Larceny project, based on the Gabriel and Gambit benchmarks.
See the results here
Running the tests
Just run
./bench <scheme> <test>
Schemes that should work:
- Bigloo (
bigloo) - Bones (
bones) - Chez (
chez) - Chibi (
chibi) - Chicken (
chicken) - Cyclone (
cyclone) - Foment (
foment) - GambitC (
gambitc) - Gauche (
gauche) - Guile (
guile) - IronScheme (
ironscheme) - Kawa (
kawa) - Larceny (
larceny) - Loko (
loko) - MIT/GNU Scheme (
mit) - Mosh (
mosh) - Petite Chez (
chez) - Picrin (
picrin) - Racket (
racket) - Rhizome/Pi (
rhizome) - RScheme (
rscheme) - Sagittarius (
sagittarius) - Scheme48 (
scheme48) - Scheme 9 from Empty Space (
s9fes) - Vicare (
vicare) - Ypsilon (
ypsilon)
This should result in a file ./results.<scheme>.
All of these Schemes are packaged for ArchLinux. The tests are run with whatever that package produces.
Run
make csv
to produce a file all.csv with the test results (<scheme>,<test>,<seconds>).
Limiting compile-time and run-time
The environment variable CPU_TIME can be set, to limit the time for compilation and for running (each individually).
CPU_TIME=3600 ./bench chez all
Path to executables
You can set an environment variable to specify the path to each Scheme.
LARCENY=/tmp/larceny/larceny MIT=/tmp/mit/mit-scheme ./bench "mit larceny" "fib ack"
Notes for specific implementations
Chicken
The chicken test code assumes the presence of
- vector-lib
- r7rs
Install them by running
chicken-install vector-lib r7rs
Racket
The racket test code assumes the presence of https://github.com/lexi-lambda/racket-r7rs.git.
Install the r7rs package installation-wide.
Run:
sudo raco pkg install --scope installation r7rs
IronScheme
Precompile system libraries (optional). It is still going to be slow.
echo "(compile-system-libraries)" | mono IronScheme.Console-v4.exe
The default binary for IronScheme (can be changed by exporting IRONSCHEME in your shell before running bench) is called ironscheme.
If you unpack IronScheme in your home, you can create a shell script like the following:
#!/bin/bash mono ~/IronScheme/IronScheme.Console-v4.exe -nologo -emacs $*
and put that somewhere in your $PATH.
Stalin
Before calling Stalin, the code is run through alexpander (by default using Chicken’s csi). Then stalin is invoked on the result.
Unsupported Schemes
Owl Lisp
Missing a lot of necessary functions, read for example.
TinyScheme
Problems with redirecting input, read chokes. No timing functions.
Safety
The benchmarking code does not use unsafe optimizations. These make code run even faster (but less safe ;) Specific options that could be used are:
- Chez
--optimize-level 3- Gerbil / Gambit
(declare (not safe))