/benchmarks

Some benchmarks of different languages

Primary LanguageMakefileMIT LicenseMIT

Table of Content

Overview

The benchmarks follow the criteria:

  • They are written as the average software developer would write them, i.e.

    • The algorithms are implemented as cited in public sources;
    • The libraries are used as described in the tutorials, documentation and examples;
    • The used data structures are idiomatic.

  • The used algorithms are similar between the languages (as the reference implementations), variants are acceptable if the reference implementation exists.

  • All final binaries are releases (optimized for performance if possible) as debug performance may vary too much depending on the compiler.

My other benchmarks: jit-benchmarks, crystal-benchmarks-game

Measurements

The measured values are:

  • time spent for the benchmark execution (loading required data and code self-testing are not measured);
  • memory consumption of the benchmark process, reported as base + increase, where base is the RSS before the benchmark and increase is the peak increase of the RSS during the benchmark;
  • energy consumption of the CPU (PP0 package) during the benchmark.

All values are presented as: median±median absolute deviation.

UPDATE: 2020-12-01

Test Cases

Brainfuck

Testing brainfuck implementations using two code samples (bench.b and mandel.b). Supports two mode:

  • Verbose (default). Prints the output immediately.
  • Quiet (if QUIET environment variable is set). Accumulates the output using Fletcher-16 checksum, and prints it out after the benchmark.

Brainfuck

bench.b

Language Time, s Memory, MiB Energy, J
C++/g++ 0.892±0.044 1.49±00.03 + 0.00±00.00 15.92±00.43
Kotlin 1.705±0.018 38.11±00.12 + 2.12±00.07 38.89±00.47
D/ldc2 1.832±0.023 3.06±00.03 + 0.00±00.00 33.17±01.19
Nim/gcc 1.860±0.036 1.92±00.04 + 0.00±00.00 35.16±02.06
C/gcc 1.887±0.034 0.50±00.00 + 0.00±00.00 36.04±00.71
Rust 1.921±0.036 2.02±00.06 + 0.00±00.00 45.72±01.08
Nim/clang 1.921±0.036 2.37±00.05 + 0.00±00.00 44.39±01.02
D/gdc 1.950±0.034 6.34±00.07 + 0.00±00.00 37.39±01.93
C/clang 2.223±0.066 0.50±00.00 + 0.00±00.00 42.64±01.37
OCaml 2.230±0.026 2.60±00.02 + 2.50±00.03 42.50±01.23
Go 2.241±0.034 3.53±00.08 + 0.00±00.00 50.55±01.43
Vala/clang 2.316±0.026 3.73±00.04 + 0.00±00.00 54.54±00.49
Crystal 2.317±0.021 3.37±00.01 + 0.00±00.00 55.93±01.28
Vala/gcc 2.335±0.062 3.77±00.04 + 0.00±00.00 47.62±01.82
C#/.NET Core 2.341±0.089 34.18±00.04 + 0.01±00.00 48.13±04.76
V/gcc 2.433±0.071 0.50±00.00 + 0.00±00.00 51.49±01.99
Java 2.458±0.113 37.49±00.09 + 0.96±00.11 48.82±02.17
Go/gccgo 2.593±0.203 20.93±00.22 + 0.00±00.00 52.43±10.28
V/clang 2.742±0.106 0.87±00.00 + 0.00±00.00 57.97±03.80
MLton 2.841±0.030 1.44±00.05 + 0.25±00.00 69.59±01.25
Chez Scheme 2.898±0.136 24.87±00.05 + 4.23±00.09 61.62±05.33
Julia 2.989±0.124 169.09±00.12 + 0.00±00.00 52.77±03.78
D/dmd 3.398±0.051 3.58±00.04 + 0.00±00.00 75.85±04.16
Scala 3.451±0.036 80.33±00.31 + 63.22±04.38 70.71±01.84
Node.js 3.876±0.091 29.72±00.07 + 2.08±00.00 86.57±02.19
C#/Mono 4.047±0.035 20.14±00.06 + 0.00±00.00 93.13±00.66
Haskell (MArray) 4.626±0.065 3.52±00.05 + 1.23±00.00 104.52±07.29
F#/.NET Core 5.759±0.060 36.41±00.05 + 90.04±00.54 122.34±02.41
Lua/luajit 6.266±0.056 2.84±00.05 + 0.00±00.00 134.09±03.45
Racket 10.241±0.263 101.02±00.05 + 0.00±00.00 188.03±09.99
Ruby/truffleruby (--jvm) 11.014±0.785 554.32±05.28 + 467.70±85.03 346.21±23.03
Python/pypy 14.634±0.483 63.77±00.19 + 45.38±00.00 348.98±16.82
Haskell 16.436±0.405 4.28±00.05 + 1.00±00.00 320.67±10.52
Ruby/truffleruby 21.492±1.470 253.22±00.14 + 638.77±14.18 517.59±37.53
Lua 57.236±1.627 2.67±00.01 + 0.00±00.00 1122.15±58.58
Ruby (--jit) 61.421±1.978 14.12±00.05 + 0.15±00.00 1122.68±34.10
Ruby 82.449±3.315 14.15±00.05 + 0.00±00.00 1706.93±130.86
Ruby/jruby 109.907±2.647 202.52±03.44 + 236.59±02.42 2259.77±110.02
Elixir 119.090±1.217 56.61±01.01 + 0.00±00.00 2268.87±68.60
Python 235.009±4.679 9.45±00.03 + 0.00±00.00 4650.00±179.37
Tcl (FP) 279.844±8.099 4.28±00.03 + 0.00±00.00 5693.86±351.71
Perl 361.937±3.237 6.51±00.04 + 0.00±00.00 7386.06±407.83
Tcl (OOP) 540.958±12.988 4.26±00.09 + 0.00±00.00 11178.46±579.19

mandel.b

Mandel in Brainfuck

Language Time, s Memory, MiB Energy, J
D/ldc2 13.023±0.141 3.02±00.04 + 0.77±00.00 282.47±15.54
C/gcc 13.024±0.044 0.49±00.00 + 1.11±00.02 232.35±06.92
C++/g++ 13.508±0.519 3.14±00.14 + 0.49±00.02 243.38±14.88
D/gdc 13.622±0.486 6.68±00.06 + 0.52±00.00 284.97±15.61
V/gcc 13.929±0.345 0.55±00.05 + 1.88±00.06 263.36±22.17
Kotlin 16.201±0.639 37.89±00.20 + 2.53±00.14 333.78±12.48
C/clang 17.205±0.296 0.54±00.00 + 1.09±00.02 409.86±07.81
C#/.NET Core 17.817±0.206 34.36±00.02 + 1.06±00.06 386.31±09.90
Nim/gcc 18.909±0.573 1.81±00.04 + 0.57±00.05 364.59±13.74
Rust 19.121±0.365 2.06±00.09 + 0.28±00.03 450.43±14.62
V/clang 19.954±0.144 0.85±00.00 + 1.97±00.09 354.30±05.02
Nim/clang 21.089±0.839 2.31±00.07 + 0.51±00.00 427.52±33.64
Vala/clang 21.997±0.558 3.54±00.04 + 2.02±00.02 417.73±08.93
Java 22.059±2.009 37.61±00.10 + 1.41±00.15 438.82±32.38
Vala/gcc 22.339±0.495 3.55±00.02 + 2.05±00.02 395.91±13.76
Scala 22.953±0.280 80.06±00.43 + 38.70±03.31 533.87±10.60
Go/gccgo 23.761±0.456 33.35±00.22 + 1.28±00.05 519.74±23.32
Crystal 23.958±0.297 3.32±00.04 + 0.45±00.02 442.85±08.69
Go 34.576±0.772 3.50±00.03 + 1.24±00.01 636.21±19.42
OCaml 36.101±0.674 3.81±00.03 + 7.19±01.04 809.54±37.56
Chez Scheme 38.361±0.223 25.47±00.03 + 3.65±00.02 928.03±21.01
D/dmd 40.256±0.116 3.48±00.03 + 0.77±00.00 890.37±04.94
C#/Mono 43.907±0.601 20.06±00.06 + 0.88±00.00 974.19±30.51
Node.js 45.577±1.511 29.36±00.12 + 6.12±00.01 911.95±74.08
MLton 52.145±0.592 1.40±00.03 + 4.11±00.00 997.68±30.34
Julia 57.873±1.982 168.56±00.17 + 0.00±00.00 1150.80±96.19
Python/pypy 65.443±0.415 64.01±00.25 + 45.97±00.09 1537.62±41.76
Haskell (MArray) 65.728±1.731 3.56±00.09 + 2.74±00.00 1333.81±82.75
Ruby/truffleruby (--jvm) 127.906±1.069 553.57±06.48 + 384.87±17.42 2850.44±62.68
F#/.NET Core 133.896±0.448 36.46±00.05 + 91.62±00.12 2663.72±08.90
Racket 167.016±3.127 100.96±00.07 + 0.00±00.00 3106.19±93.72
Ruby/truffleruby 169.044±5.202 252.82±00.38 + 661.55±17.47 3637.07±123.52
Haskell 224.213±5.516 3.65±00.06 + 26.29±00.00 4739.08±376.63
Lua/luajit 246.285±3.792 2.79±00.03 + 0.86±00.00 4889.38±291.91

Base64

Testing base64 encoding/decoding of the large blob into the newly allocated buffers.

Base64

Language Time, s Memory, MiB Energy, J
C/gcc (aklomp) 0.156±0.002 1.90±00.02 + 0.00±00.00 3.43±00.14
C/gcc 1.190±0.010 1.87±00.04 + 0.00±00.00 24.92±00.52
Rust 1.246±0.045 2.49±00.08 + 0.01±00.01 28.21±00.74
V/clang 1.522±0.026 1.98±00.04 + 0.47±00.01 32.63±00.77
V/gcc 1.624±0.026 1.73±00.03 + 0.23±00.11 35.31±01.20
D/ldc2 1.745±0.011 3.40±00.02 + 3.66±00.00 30.74±00.31
Nim/clang 1.877±0.024 2.68±00.06 + 4.41±00.03 33.76±01.21
Nim/gcc 2.048±0.075 2.23±00.05 + 4.44±00.00 40.58±03.26
Crystal 2.100±0.006 3.76±00.03 + 1.79±00.02 43.08±00.46
D/gdc 2.137±0.015 7.05±00.09 + 3.46±00.03 37.92±01.38
Java 2.248±0.032 38.45±00.09 + 360.35±13.12 49.02±02.22
Ruby 2.336±0.029 14.41±00.05 + 42.64±00.90 53.21±02.22
Ruby (--jit) 2.364±0.048 14.47±00.04 + 43.78±00.06 53.65±03.10
Kotlin 2.411±0.058 39.43±00.08 + 339.98±11.43 51.45±02.95
Go 2.497±0.007 4.52±00.13 + 4.89±00.10 47.17±00.43
Scala 2.518±0.028 81.21±00.46 + 64.54±07.62 55.40±03.45
C++/g++ (libcrypto) 2.623±0.028 5.34±00.04 + 0.07±00.00 47.14±00.51
PHP 2.945±0.031 15.72±00.15 + 0.00±00.00 51.14±01.07
Perl (MIME::Base64) 3.024±0.046 14.16±00.03 + 0.02±00.00 52.92±01.33
Node.js 3.065±0.030 29.89±00.05 + 1029.92±00.07 67.19±00.76
Go/gccgo 3.660±0.012 22.07±00.12 + 7.22±00.22 72.73±00.40
D/dmd 3.663±0.066 3.67±00.06 + 3.67±00.00 74.06±00.79
Tcl 4.295±0.072 5.06±00.05 + 0.00±00.00 73.76±01.81
Python/pypy 4.723±0.117 64.13±00.15 + 45.72±00.05 101.07±05.88
C#/.NET Core 5.213±0.062 34.44±00.09 + 41.21±01.20 97.46±00.94
Python 5.448±0.112 9.21±00.05 + 0.18±00.00 102.56±05.80
Julia 5.857±0.093 200.32±00.21 + 42.88±00.21 111.74±01.76
Ruby/truffleruby (--jvm) 5.859±0.142 551.20±02.74 + 399.55±28.65 124.74±07.81
C#/Mono 7.565±0.204 20.60±00.05 + 18.49±00.06 144.66±05.48
Ruby/jruby 9.862±0.279 192.07±03.01 + 173.11±05.41 213.68±16.56
Perl (MIME::Base64::Perl) 16.414±0.631 15.46±00.05 + 0.20±00.08 353.95±27.54
Ruby/truffleruby 20.647±0.721 253.00±00.31 + 404.70±00.60 413.32±37.94

Json

Testing parsing and simple calculating of values from a big JSON file.

Json

Language Time, s Memory, MiB Energy, J
C++/g++ (DAW JSON Link) 0.082±0.002 109.27±00.02 + 0.00±00.00 1.95±00.06
C++/g++ (simdjson On-Demand) 0.088±0.001 109.90±00.05 + 59.81±00.12 1.69±00.08
D/gdc (fast) 0.100±0.001 219.98±00.03 + 11.12±00.03 2.58±00.06
Rust (Serde Custom) 0.141±0.001 108.43±00.09 + 0.00±00.00 2.35±00.04
Rust (Serde Typed) 0.150±0.001 108.48±00.07 + 11.51±00.03 2.52±00.04
C++/g++ (gason) 0.158±0.004 109.25±00.01 + 97.17±00.03 3.19±00.21
C++/g++ (simdjson DOM) 0.160±0.003 109.81±00.03 + 176.60±00.00 3.21±00.08
C++/g++ (RapidJSON) 0.229±0.006 109.27±00.02 + 128.82±00.00 4.51±00.35
C++/g++ (Boost.JSON) 0.481±0.009 109.80±00.03 + 435.70±00.00 11.83±00.26
Java 0.528±0.018 252.65±00.15 + 75.46±01.57 14.53±00.36
C++/g++ (RapidJSON SAX) 0.534±0.021 109.46±00.02 + 0.00±00.00 11.52±00.82
Scala 0.609±0.009 327.43±00.58 + 73.55±00.87 16.90±00.72
Julia (JSON3) 0.615±0.031 355.85±00.44 + 243.86±02.20 13.47±01.25
Node.js 0.663±0.010 242.73±00.08 + 185.38±00.82 14.04±00.37
Go (jsoniter) 0.702±0.019 224.79±00.11 + 13.89±00.15 14.43±01.07
Python/pypy 0.780±0.013 277.41±00.16 + 127.96±00.00 18.71±00.61
Rust (Serde Untyped) 0.846±0.013 108.48±00.08 + 839.98±00.00 19.09±00.42
Crystal (Pull) 0.856±0.033 110.30±00.05 + 18.20±00.04 16.65±01.40
Crystal (Schema) 0.883±0.039 110.36±00.03 + 47.05±00.08 15.84±01.53
C#/.NET Core (System.Text.Json) 0.982±0.018 465.50±00.22 + 135.59±00.10 22.84±00.75
Perl (Cpanel::JSON::XS) 1.000±0.013 121.32±00.06 + 402.72±00.00 18.21±00.37
Go 1.129±0.049 113.94±00.19 + 95.95±00.02 22.35±01.99
Crystal 1.147±0.029 110.37±00.03 + 393.35±00.03 24.00±01.31
PHP 1.204±0.012 121.69±00.09 + 682.01±00.00 25.92±00.20
V/gcc 1.211±0.024 107.42±00.01 + 484.49±00.03 27.41±00.83
V/clang 1.299±0.021 107.91±00.03 + 484.48±00.03 29.13±01.05
Go/gccgo 1.401±0.025 132.57±00.07 + 95.88±00.20 31.66±01.06
Nim/gcc (Packedjson) 1.458±0.035 108.80±00.04 + 290.55±00.00 31.04±03.08
Nim/clang (Packedjson) 1.459±0.028 109.22±00.08 + 290.55±00.00 34.02±00.59
C++/g++ (json-c) 1.561±0.044 109.42±00.01 + 1216.08±00.00 30.31±01.55
Clojure 1.561±0.053 491.24±08.10 + 516.55±22.59 42.27±01.67
Haskell 1.701±0.024 4.41±00.03 + 5.42±00.03 39.66±00.56
C#/.NET Core 1.736±0.054 474.19±00.09 + 288.64±00.06 30.96±01.72
Nim/gcc 1.756±0.020 108.60±00.11 + 915.75±00.06 39.94±01.11
Python 1.757±0.056 115.88±00.05 + 377.23±00.01 38.02±02.48
CPython (UltraJSON) 1.771±0.020 117.52±00.04 + 545.45±00.90 35.77±00.50
Nim/clang 1.880±0.064 109.09±00.10 + 915.65±00.10 37.34±03.24
C#/Mono 2.072±0.048 462.61±00.08 + 0.18±00.00 45.96±02.08
Ruby (YAJL) 2.183±0.061 120.60±00.03 + 286.98±00.01 50.23±02.52
D/gdc 2.234±0.115 113.23±00.03 + 600.32±00.00 44.22±05.07
Ruby 2.260±0.067 120.71±00.04 + 277.85±00.04 42.18±01.80
Ruby (--jit) 2.316±0.029 120.76±00.02 + 277.83±00.05 44.52±01.57
D/ldc2 2.503±0.033 109.55±00.05 + 680.07±00.04 50.98±01.91
Rust (jq) 3.578±0.063 110.41±00.05 + 774.34±00.26 79.25±01.55
Ruby/jruby 3.815±0.041 473.21±08.23 + 1448.93±24.01 116.78±04.54
C++/g++ (Boost.PropertyTree) 4.353±0.127 109.59±00.02 + 1440.00±00.00 94.67±07.97
D/dmd 4.867±0.131 110.06±00.04 + 680.07±00.03 101.12±04.66
Perl (JSON::Tiny) 11.508±0.117 121.91±00.06 + 525.12±00.04 268.69±04.33
Ruby/truffleruby (--jvm) 101.930±0.923 730.89±03.57 + 1288.83±43.61 2876.01±27.44
Ruby/truffleruby 582.209±8.945 731.12±00.18 + 2184.33±38.93 11738.06±202.59

Matmul

Testing allocating and multiplying matrices.

Matmul

Language Time, s Memory, MiB Energy, J
D/ldc2 (lubeck) 0.069±0.001 6.70±00.04 + 56.29±00.19 3.83±00.09
Nim/clang (Arraymancer) 0.070±0.001 5.54±00.04 + 57.67±00.02 4.02±00.04
Nim/gcc (Arraymancer) 0.079±0.001 5.53±00.04 + 57.67±00.02 4.70±00.09
Python (NumPy) 0.105±0.002 27.06±00.06 + 57.64±00.04 5.91±00.21
Julia (threads: 8) 0.190±0.003 222.04±00.19 + 52.41±00.12 11.08±00.30
Java (ND4J) 0.206±0.026 137.15±03.95 + 87.35±00.00 9.96±01.00
Julia (threads: 1) 0.531±0.019 221.56±00.17 + 53.05±00.22 12.27±00.94
D/ldc2 1.965±0.010 3.55±00.04 + 70.11±00.00 43.78±00.60
D/gdc 2.072±0.027 6.68±00.05 + 70.70±00.01 48.59±01.94
D/dmd 2.103±0.009 3.75±00.11 + 70.43±00.13 48.29±00.94
Java 3.289±0.020 37.92±00.04 + 77.27±00.26 74.62±01.52
C/gcc 3.320±0.010 2.03±00.02 + 68.06±00.00 76.08±01.58
Rust 3.363±0.008 2.62±00.06 + 68.32±00.00 69.32±01.58
Nim/gcc 3.434±0.017 2.63±00.03 + 74.77±04.64 70.32±01.18
Scala 3.451±0.039 84.70±00.88 + 80.50±03.14 80.20±01.58
Nim/clang 3.454±0.009 3.08±00.06 + 72.19±02.19 71.54±01.32
Julia (no BLAS) 3.503±0.024 177.98±00.14 + 69.74±00.07 74.83±00.82
Go 3.572±0.037 3.79±00.08 + 73.24±00.23 76.44±01.29
Go/gccgo 3.604±0.055 27.67±06.19 + 72.72±00.21 75.21±00.74
Crystal 3.623±0.010 4.13±00.04 + 59.70±00.01 90.24±00.72
V/clang 3.642±0.054 2.44±00.01 + 68.84±00.00 90.91±01.22
Swift 3.657±0.059 148.95±00.13 + 59.76±00.07 87.78±02.97
Node.js 3.891±0.137 33.06±00.01 + 71.18±00.03 82.84±05.76
Kotlin 3.990±0.016 37.62±00.11 + 77.96±00.23 69.54±02.71
V/gcc 4.671±0.152 1.97±00.05 + 68.84±00.00 97.06±09.17
Python/pypy 6.261±0.256 64.29±00.20 + 69.20±00.03 122.94±12.29
C#/.NET Core 6.892±0.269 33.88±00.14 + 69.12±00.01 150.17±12.02
C#/Mono 10.532±0.078 20.20±00.05 + 69.02±00.01 237.07±02.79
Ruby/truffleruby 38.623±0.297 527.69±00.21 + 482.24±03.58 706.62±15.54
Ruby/truffleruby (--jvm) 70.781±1.848 597.32±03.94 + 330.89±08.63 1912.53±68.70
Ruby (--jit) 213.343±7.607 15.17±00.04 + 69.09±00.00 4581.51±99.85
Ruby 214.756±3.892 15.14±00.04 + 68.89±00.00 4585.32±103.55
Python 240.588±5.667 9.84±00.04 + 68.58±00.00 5021.18±181.03
Tcl 348.604±8.750 7.25±00.03 + 400.41±00.03 7605.35±576.78
Perl 402.099±5.437 8.96±00.08 + 599.69±00.05 8531.93±821.67
Ruby/jruby 488.395±11.851 279.21±05.66 + 658.86±07.14 10808.74±390.65

Tests Execution

Environment

CPU: Intel(R) Core(TM) i7-10710U

Base Docker image: Debian GNU/Linux bullseye/sid

Language Version
.NET Core 5.0.100
C#/.NET Core 3.8.0-5.20519.18 (4c195c3a)
C#/Mono 6.12.0.90
C/clang 11.0.0
C/gcc 10.2.0
Chez Scheme 9.5.4
Clojure "1.10.1"
Crystal 0.35.1
D/dmd v2.094.2
D/gdc 10.2.0
D/ldc2 1.24.0
Elixir 1.10.3
F#/.NET Core 11.0.0.0 for F# 5.0
Go go1.15.5
Go/gccgo 10.2.0
Haskell 8.10.2
Java 15.0.1
Julia v"1.5.3"
Kotlin 1.4.20
Lua Lua 5.4
Lua/luajit LuaJIT 2.1.0-beta3
MLton 20201002
Nim 1.4.0
Node.js v15.3.0
OCaml 4.11.1
PHP 7.4.11
Perl v5.32.0
Python 3.8.6
Python/pypy 7.3.3-beta0 for Python 3.7.9
Racket "7.9"
Ruby 2.7.2p137
Ruby/jruby 9.2.13.0
Ruby/truffleruby 20.3.0
Rust 1.48.0
Scala 2.13.4
Swift swift-5.3.1-RELEASE
Tcl 8.6
V 0.1.30
Vala 0.48.12

Using Docker

Build the image:

$ docker build docker/ -t benchmarks

Run the image:

$ docker run -it --rm -v $(pwd):/src benchmarks <cmd>

where is:

  • versions (print installed language versions);
  • shell (start the shell);
  • brainfuck bench (build and run Brainfuck bench.b benchmarks);
  • brainfuck mandel (build and run Brainfuck mandel.b benchmarks);
  • base64 (build and run Base64 benchmarks);
  • json (build and run Json benchmarks);
  • matmul (build and run Matmul benchmarks);

Please note that the actual measurements provided in the project are taken semi-manually (via shell) as the full update takes days and could have occassional issues in Docker.

There is a Makefile that could be used to simlify Docker usage:

  • make build (build the image);
  • make versions (run the image with the versions command);
  • make shell (run the image with the `shell' command);
  • make toc (utility rule to update ToC in this README, requires git-markdown-toc available in PATH).

Please note that the make shell rule requires cpupower utility installed that is invoked with sudo to set cpufreq's performance governon (it runs the CPU at the maximum frequence to eliminate throttling issues).

Manual Execution

Makefiles contain recipes for building and executing tests with the proper dependencies. Please use make run (and make run2 where applicable). The measurements are taken using analyze.rb script:

$ cd <test suite>
$ ../analyze.rb make run
$ ../analyze.rb make run[<single test>]

Please note that the measurements could take hours. It uses 10 iterations by default, but it could be changed using ATTEMPTS environment variable:

$ ATTEMPTS=1 ../analyze.rb make run

Prerequisites

Please use Dockerfile as a reference regarding which packages and tools are required.

For all (optional):

  • Powercap for reading energy counters in Linux (Debian package powercap-utils).

For Python:

  • NumPy for matmul tests (Debian package python3-numpy).
  • UltraJSON for JSON tests (Debian package python3-ujson).

For C++:

  • Boost for JSON tests (Debian package libboost-dev).
  • JSON-C for JSON tests (Debian package libjson-c-dev).

For Rust:

  • libjq for jq test (Debian packages libjq-dev, libonig-dev and environment variable JQ_LIB_DIR=/usr/lib/x86_64-linux-gnu/).

For Java, Scala:

  • Coursier for downloading Maven artifacts.

For Lua:

  • LuaRocks for installing dependencies (Debian package luarocks).

For Haskell:

  • network for TCP connectivity between the tests and the test runner.
  • raw-strings-qq for raw string literals used in tests.

For Perl:

  • cpanminus for installing modules from CPAN (Debian package cpanminus).

Contribution

Please follow the criteria specified in the overview. Besides that please ensure that the communication protocol between a test and the test runner is satisfied:

  • The test runner listens on localhost:9001;
  • All messages are sent using TCP sockets closed immediately after the message has been sent;
  • There are two messages sent from a test (it establishes the measurement boundary):
    1. The beginning message having the format name of the test/tprocess ID (the process ID is used to measure the memory consumption). Please note that the name of the test couldn't use Tab character as it's a delimiter;
    2. The end message with any content (mostly it's "stop" for consistency).
  • The test runner could be unavailable (if the test is launched as is) and the test should gracefully handle it.

Makefile guide

Binary executables

If the test is compiled into a single binary, then two sections of the Makefile require changes:

  • append a new target (the final binary location) into executables variable;
  • append the proper target rule.

Compiled artifacts

If the test is compiled, but can't be executed directly as a binary, then three sections of the Makefile require changes:

  • append a new target (the final artifact location) into artifacts variable;
  • append the proper target rule to compile the test;
  • append run[<target_artifact>] rule to run the test.

Scripting language

If the test doesn't require compilation, then two sections of the Makefile requires changes:

  • append run[<script_file>] into all_runners variable;
  • append run[<script_file>] rule to run the test.