/benchmarks

Some benchmarks of different languages

Primary LanguageMakefileMIT LicenseMIT

Table of Content

Overview

The benchmarks follow the criteria:

  • They are written as the average software developer would write them, i.e.

    • The algorithms are implemented as cited in public sources;
    • The libraries are used as described in the tutorials, documentation and examples;
    • The used data structures are idiomatic.

  • The used algorithms are similar between the languages (as the reference implementations), variants are acceptable if the reference implementation exists.

  • All final binaries are releases (optimized for performance if possible) as debug performance may vary too much depending on the compiler.

My other benchmarks: jit-benchmarks, crystal-benchmarks-game

Measurements

The measured values are:

  • time spent for the benchmark execution (loading required data and code self-testing are not measured);
  • memory consumption of the benchmark process, reported as base + increase, where base is the RSS before the benchmark and increase is the peak increase of the RSS during the benchmark;
  • energy consumption of the CPU (PP0 package) during the benchmark.

All values are presented as: median±median absolute deviation.

UPDATE: 2021-01-11

Test Cases

Brainfuck

Testing brainfuck implementations using two code samples (bench.b and mandel.b). Supports two mode:

  • Verbose (default). Prints the output immediately.
  • Quiet (if QUIET environment variable is set). Accumulates the output using Fletcher-16 checksum, and prints it out after the benchmark.

Brainfuck

bench.b

Language Time, s Memory, MiB Energy, J
C++/g++ 0.887±0.042 1.49±00.00 + 0.00±00.00 17.29±01.73
Nim/gcc 1.786±0.034 1.86±00.07 + 0.00±00.00 39.74±01.19
D/ldc2 1.854±0.038 3.01±00.02 + 0.00±00.00 33.68±01.80
Kotlin 1.865±0.034 38.20±00.29 + 1.98±00.26 32.87±00.35
C/gcc 1.907±0.065 0.50±00.01 + 0.00±00.00 35.03±00.84
Nim/clang 1.945±0.045 2.32±00.07 + 0.00±00.00 45.73±01.24
D/gdc 1.995±0.061 6.33±00.06 + 0.00±00.00 36.50±01.89
Rust 2.119±0.047 1.98±00.07 + 0.00±00.00 38.02±00.97
OCaml 2.130±0.094 2.57±00.03 + 2.50±00.05 49.06±04.11
C/clang 2.260±0.051 0.50±00.00 + 0.00±00.00 43.96±02.89
C#/.NET Core 2.311±0.085 34.37±00.06 + 0.01±00.00 48.42±04.26
Go 2.329±0.092 3.51±00.05 + 0.00±00.00 48.09±03.76
Vala/gcc 2.345±0.095 3.73±00.03 + 0.00±00.00 46.38±02.41
Crystal 2.388±0.096 3.35±00.06 + 0.00±00.00 53.85±00.83
Vala/clang 2.408±0.079 3.75±00.03 + 0.00±00.00 54.26±01.53
V/gcc 2.439±0.086 0.52±00.03 + 0.00±00.00 52.32±02.56
Java 2.502±0.144 37.73±00.15 + 0.91±00.18 47.92±02.98
Go/gccgo 2.586±0.238 21.28±00.19 + 0.00±00.00 53.03±07.57
V/clang 2.752±0.106 0.87±00.00 + 0.00±00.00 58.64±04.35
Julia 2.942±0.135 169.08±00.07 + 0.00±00.00 55.03±06.16
Chez Scheme 2.950±0.121 24.91±00.02 + 4.20±00.03 61.04±06.15
MLton 2.980±0.131 1.44±00.02 + 0.25±00.00 63.08±06.69
Scala 3.503±0.029 80.33±00.45 + 58.71±03.87 69.07±02.62
D/dmd 3.636±0.065 3.52±00.07 + 0.00±00.00 64.71±03.91
C#/Mono 4.053±0.021 20.24±00.05 + 0.00±00.00 91.30±03.42
Node.js 4.200±0.194 30.67±00.09 + 3.62±00.01 74.58±03.73
Haskell (MArray) 4.672±0.068 3.69±00.07 + 1.16±00.00 84.51±01.79
F#/.NET Core 5.630±0.033 36.51±00.05 + 90.36±00.13 130.47±01.40
Lua/luajit 6.762±0.158 2.89±00.05 + 0.00±00.00 114.01±03.57
Racket 9.654±0.103 101.03±00.09 + 0.00±00.00 215.06±03.58
Ruby/truffleruby (--jvm) 10.676±0.709 553.73±07.23 + 425.15±55.94 337.56±09.05
Python/pypy 15.303±0.588 63.97±00.09 + 45.37±00.00 339.70±27.42
Haskell 16.627±0.594 3.82±00.04 + 0.88±00.00 317.75±16.26
Ruby/truffleruby 21.569±1.239 253.01±00.44 + 646.27±09.10 465.69±119.53
Lua 59.453±1.091 2.67±00.02 + 0.00±00.00 1086.03±14.38
Ruby (--jit) 61.091±1.388 14.05±00.03 + 0.23±00.00 1104.23±29.71
Ruby 88.905±3.702 14.03±00.06 + 0.00±00.00 1874.14±104.91
Ruby/jruby 111.976±4.390 199.00±04.69 + 232.92±01.53 2283.61±73.13
Elixir 115.244±2.160 56.56±00.51 + 0.00±00.00 2538.04±130.27
Python 232.090±5.249 10.41±00.03 + 0.00±00.00 5198.30±150.90
Tcl (FP) 277.932±7.323 4.28±00.04 + 0.00±00.00 6132.28±205.33
Perl 367.572±10.403 6.48±00.03 + 0.00±00.00 6994.56±109.33
Tcl (OOP) 562.710±8.313 4.28±00.05 + 0.00±00.00 10516.03±111.12

mandel.b

Mandel in Brainfuck

Language Time, s Memory, MiB Energy, J
C/gcc 13.327±0.268 0.55±00.01 + 1.07±00.02 226.88±06.84
C++/g++ 13.353±0.246 3.27±00.05 + 0.47±00.00 271.19±05.91
D/gdc 13.611±0.159 6.74±00.10 + 0.52±00.00 290.12±04.00
D/ldc2 14.124±0.330 3.04±00.03 + 0.77±00.00 241.94±04.27
Nim/gcc 15.135±0.699 1.87±00.06 + 0.51±00.00 285.26±21.64
V/gcc 15.619±0.648 0.56±00.00 + 1.95±00.06 310.08±24.89
Kotlin 16.558±0.253 38.27±00.23 + 2.63±00.31 321.66±27.02
C/clang 17.680±0.169 0.52±00.02 + 1.09±00.03 404.60±02.14
C#/.NET Core 19.262±0.280 34.42±00.04 + 1.04±00.03 332.74±13.85
V/clang 20.233±0.836 0.98±00.12 + 1.95±00.09 391.71±32.22
Rust 20.594±0.390 1.97±00.04 + 0.25±00.00 389.28±16.81
Vala/clang 21.297±0.454 3.59±00.03 + 2.04±00.05 476.19±15.96
Vala/gcc 21.454±0.172 3.59±00.03 + 2.03±00.03 454.96±11.26
Nim/clang 21.827±0.352 2.31±00.01 + 0.51±00.00 398.81±09.98
Java 23.685±1.904 37.70±00.22 + 1.41±00.12 458.21±31.50
Crystal 23.903±0.971 3.35±00.03 + 0.46±00.02 475.53±36.59
Scala 24.425±0.438 80.31±00.49 + 41.52±03.52 472.17±28.71
Go/gccgo 26.150±0.487 21.30±00.05 + 1.28±00.01 448.70±12.23
Go 33.741±0.576 3.53±00.05 + 1.25±00.01 709.36±28.06
OCaml 37.524±1.155 3.88±00.03 + 9.40±01.92 754.09±61.18
Chez Scheme 39.908±0.185 25.44±00.02 + 3.65±00.02 900.93±06.60
C#/Mono 44.691±0.148 20.34±00.05 + 0.89±00.00 963.56±10.00
D/dmd 46.028±0.501 3.55±00.08 + 0.77±00.00 821.16±10.79
Node.js 47.273±0.664 30.62±00.10 + 6.18±00.01 862.48±25.11
MLton 51.157±0.803 1.44±00.05 + 4.11±00.00 1125.53±16.85
Julia 57.933±0.971 169.52±00.18 + 0.00±00.00 1217.32±17.16
Haskell (MArray) 61.182±0.580 3.68±00.05 + 2.68±00.00 1366.02±31.06
Python/pypy 68.463±1.458 64.10±00.19 + 46.16±00.09 1507.60±67.16
Ruby/truffleruby (--jvm) 130.745±1.922 556.91±06.97 + 399.51±20.62 2730.87±29.04
F#/.NET Core 131.618±0.400 36.52±00.12 + 91.49±00.04 2979.33±18.09
Racket 163.988±1.458 101.01±00.03 + 0.00±00.00 3487.58±19.89
Ruby/truffleruby 173.815±2.968 252.98±00.33 + 652.14±18.65 3763.90±85.58
Haskell 217.835±1.549 3.87±00.02 + 26.16±00.00 4856.01±69.73
Lua/luajit 252.222±3.794 2.96±00.07 + 0.86±00.00 4624.78±140.51

Base64

Testing base64 encoding/decoding of the large blob into the newly allocated buffers.

Base64

Language Time, s Memory, MiB Energy, J
C/gcc (aklomp) 0.155±0.004 1.91±00.01 + 0.00±00.00 3.53±00.22
C/gcc 1.286±0.022 1.88±00.01 + 0.00±00.00 21.62±00.47
Rust 1.367±0.018 2.43±00.08 + 0.04±00.03 23.76±00.76
V/gcc 1.490±0.017 1.72±00.04 + 0.27±00.03 31.92±00.56
V/clang 1.547±0.044 2.01±00.04 + 0.48±00.01 32.47±01.29
Nim/clang 1.776±0.069 2.75±00.05 + 4.38±00.00 35.06±03.11
D/ldc2 1.776±0.009 3.46±00.08 + 3.67±00.00 30.96±00.26
Crystal 1.970±0.004 3.81±00.03 + 1.81±00.01 47.33±00.28
D/gdc 1.991±0.013 7.12±00.03 + 3.46±00.00 43.91±00.53
Nim/gcc 2.127±0.066 2.25±00.04 + 4.47±00.03 37.94±01.81
Java 2.221±0.047 38.51±00.11 + 328.86±19.54 48.33±02.29
Ruby 2.320±0.080 14.45±00.04 + 57.18±01.10 49.27±03.59
Kotlin 2.401±0.042 39.44±00.15 + 314.60±02.85 54.82±00.96
Ruby (--jit) 2.434±0.033 14.44±00.05 + 56.54±01.23 46.07±01.63
Go 2.526±0.007 4.57±00.06 + 4.86±00.27 47.73±00.46
Scala 2.590±0.055 80.96±00.24 + 70.38±06.02 50.33±00.82
C++/g++ (libcrypto) 2.662±0.042 5.46±00.04 + 0.07±00.00 47.36±01.47
PHP 2.775±0.065 15.77±00.13 + 0.00±00.00 59.73±02.12
Node.js 3.039±0.035 31.17±00.04 + 1029.97±00.12 65.58±01.74
Perl (MIME::Base64) 3.062±0.072 14.14±00.08 + 0.02±00.00 53.75±02.24
Go/gccgo 3.530±0.003 22.04±00.10 + 7.43±00.21 74.98±01.12
D/dmd 3.940±0.104 3.77±00.03 + 3.67±00.06 78.69±03.66
Python 3.989±0.034 10.12±00.03 + 0.18±00.00 90.21±00.87
Tcl 4.167±0.146 5.17±00.08 + 0.00±00.00 83.77±06.55
Python/pypy 4.769±0.055 63.84±00.14 + 45.68±00.03 104.57±01.58
C#/.NET Core 5.330±0.017 34.62±00.04 + 36.88±04.38 97.45±01.70
Julia 5.715±0.120 201.46±00.06 + 42.97±00.08 131.81±01.76
Ruby/truffleruby (--jvm) 6.023±0.110 555.55±06.98 + 324.02±32.15 120.70±05.32
C#/Mono 7.229±0.042 20.80±00.06 + 18.48±00.07 170.53±01.78
Ruby/jruby 10.802±0.487 201.17±07.47 + 164.91±12.64 216.26±18.52
Perl (MIME::Base64::Perl) 16.743±0.684 15.41±00.08 + 0.26±00.02 344.13±23.56
Ruby/truffleruby 20.875±0.745 253.25±01.02 + 404.54±00.39 405.75±37.59

Json

Testing parsing and simple calculating of values from a big JSON file.

Json

Language Time, s Memory, MiB Energy, J
C++/g++ (DAW JSON Link) 0.090±0.004 109.26±00.02 + 0.00±00.00 2.04±00.21
C++/g++ (simdjson On-Demand) 0.092±0.003 109.83±00.02 + 59.55±00.00 1.97±00.12
D/gdc (fast) 0.105±0.004 220.09±00.01 + 11.27±00.26 2.49±00.14
Rust (Serde Custom) 0.138±0.006 108.52±00.04 + 0.00±00.00 2.85±00.28
Rust (Serde Typed) 0.150±0.006 108.44±00.06 + 11.89±00.13 2.96±00.19
C++/g++ (gason) 0.160±0.006 109.25±00.02 + 97.17±00.06 3.61±00.30
C++/g++ (simdjson DOM) 0.165±0.003 109.87±00.06 + 176.60±00.00 3.28±00.14
C++/g++ (RapidJSON) 0.232±0.008 109.26±00.02 + 128.85±00.03 4.56±00.24
C++/g++ (Boost.JSON) 0.495±0.020 109.85±00.05 + 435.70±00.00 11.05±01.15
Java 0.533±0.020 252.48±00.10 + 74.44±01.27 13.90±00.59
C++/g++ (RapidJSON SAX) 0.570±0.022 109.50±00.03 + 0.00±00.00 10.43±00.53
Julia (JSON3) 0.597±0.015 348.38±00.08 + 256.97±01.51 13.68±00.48
Scala 0.610±0.018 320.42±06.88 + 73.82±00.73 16.67±00.71
Node.js 0.663±0.011 244.09±00.09 + 184.99±00.20 14.68±00.33
Go (jsoniter) 0.691±0.017 224.87±00.16 + 13.79±00.21 15.68±00.90
Python/pypy 0.831±0.023 277.41±00.23 + 127.96±00.00 16.15±00.48
V/clang 0.841±0.024 107.97±00.11 + 484.45±00.09 15.29±00.24
Crystal (Pull) 0.843±0.004 110.35±00.05 + 18.19±00.05 17.42±00.37
Rust (Serde Untyped) 0.850±0.013 108.34±00.06 + 840.04±00.03 18.99±01.02
V/gcc 0.895±0.026 107.39±00.01 + 484.56±00.03 19.52±01.73
Crystal (Schema) 0.917±0.031 110.27±00.03 + 47.04±00.07 14.97±00.54
Perl (Cpanel::JSON::XS) 0.993±0.030 121.27±00.07 + 402.72±00.00 19.53±00.70
C#/.NET Core (System.Text.Json) 1.065±0.016 465.55±00.23 + 135.54±00.11 19.24±00.29
Crystal 1.142±0.027 110.36±00.02 + 393.34±00.03 25.01±00.65
Go 1.167±0.037 113.96±00.11 + 95.70±00.03 21.04±00.71
PHP 1.191±0.010 121.61±00.19 + 682.01±00.00 26.72±00.27
Go/gccgo 1.409±0.008 132.28±00.16 + 95.87±00.21 31.98±00.53
C++/g++ (json-c) 1.466±0.018 109.45±00.01 + 1216.08±00.00 35.13±00.58
Nim/gcc (Packedjson) 1.522±0.037 108.77±00.06 + 290.55±00.00 27.16±00.46
Nim/clang (Packedjson) 1.528±0.057 109.02±00.09 + 290.81±00.00 31.61±02.83
Clojure 1.610±0.061 487.96±06.62 + 520.30±04.81 41.80±02.48
Python 1.721±0.010 116.67±00.03 + 377.21±00.00 40.79±00.48
C#/.NET Core 1.729±0.060 474.64±00.30 + 288.63±00.08 31.25±01.48
Haskell 1.741±0.016 4.73±00.07 + 4.44±00.03 41.18±01.13
CPython (UltraJSON) 1.774±0.016 118.31±00.02 + 545.93±01.85 38.65±01.38
Nim/clang 1.826±0.025 109.05±00.07 + 919.62±00.06 42.11±01.44
Nim/gcc 1.865±0.071 108.61±00.16 + 919.52±00.16 38.84±03.13
C#/Mono 2.152±0.105 462.45±00.13 + 0.17±00.01 43.70±04.37
Ruby 2.198±0.021 120.60±00.02 + 410.64±00.04 50.74±00.63
D/gdc 2.243±0.092 113.30±00.02 + 600.25±00.00 50.49±02.53
Ruby (--jit) 2.290±0.064 120.62±00.06 + 410.77±00.06 49.99±04.26
Ruby (YAJL) 2.307±0.079 120.51±00.03 + 281.59±00.00 51.52±02.14
D/ldc2 2.532±0.040 109.59±00.04 + 680.19±00.02 50.59±01.78
Rust (jq) 3.496±0.125 110.42±00.06 + 775.88±00.78 78.27±02.30
Ruby/jruby 3.841±0.114 475.22±07.28 + 1481.21±39.95 113.68±03.22
C++/g++ (Boost.PropertyTree) 4.302±0.080 109.65±00.05 + 1440.06±00.00 102.62±01.70
D/dmd 4.951±0.083 110.10±00.06 + 680.20±00.01 102.50±03.04
Perl (JSON::Tiny) 12.346±0.529 121.91±00.04 + 525.10±00.01 236.77±15.96
Ruby/truffleruby (--jvm) 103.399±0.732 725.13±06.18 + 1191.03±96.08 2883.54±44.89
Ruby/truffleruby 585.759±4.354 731.19±00.36 + 2023.45±42.56 12242.31±167.47

Matmul

Testing allocating and multiplying matrices.

Matmul

Language Time, s Memory, MiB Energy, J
D/ldc2 (lubeck) 0.067±0.002 6.74±00.06 + 55.61±00.16 3.78±00.08
Python (NumPy) 0.102±0.001 27.59±00.10 + 57.67±00.08 5.40±00.13
Nim/gcc (Arraymancer) 0.157±0.016 5.63±00.07 + 57.68±00.01 8.35±00.54
Nim/clang (Arraymancer) 0.166±0.022 6.44±00.12 + 57.42±00.04 7.92±00.63
Julia (threads: 8) 0.200±0.005 221.80±00.12 + 52.58±00.17 10.40±00.40
Java (ND4J) 0.206±0.028 135.38±01.86 + 87.35±00.00 9.81±01.38
Julia (threads: 1) 0.537±0.012 221.39±00.14 + 53.25±00.03 12.99±00.29
D/ldc2 1.984±0.013 3.56±00.02 + 70.11±00.00 44.09±00.55
D/gdc 2.118±0.049 6.65±00.07 + 70.71±00.01 49.15±02.42
D/dmd 2.146±0.035 3.54±00.04 + 70.11±00.00 45.76±01.20
C/gcc 3.307±0.016 2.02±00.04 + 68.06±00.00 77.93±00.61
Java 3.331±0.044 37.95±00.10 + 77.51±00.20 76.41±01.37
Rust 3.389±0.016 2.63±00.06 + 68.32±00.00 71.84±00.43
Scala 3.422±0.107 84.64±00.68 + 79.79±02.91 77.24±01.05
Nim/clang 3.462±0.014 3.11±00.02 + 70.64±00.52 72.49±00.79
Nim/gcc 3.465±0.014 2.61±00.03 + 71.41±01.29 71.65±01.21
Go/gccgo 3.560±0.016 21.43±00.20 + 72.99±00.18 77.74±01.42
Julia (no BLAS) 3.560±0.085 177.86±00.10 + 69.76±00.04 72.02±02.93
Go 3.568±0.018 3.80±00.03 + 73.18±00.27 79.64±01.43
V/clang 3.707±0.041 2.44±00.02 + 68.84±00.00 89.45±02.24
Crystal 3.815±0.145 4.17±00.02 + 59.63±00.03 77.84±04.22
Swift 3.831±0.107 148.70±00.06 + 59.66±00.11 76.12±04.76
Node.js 3.856±0.067 34.16±00.12 + 71.25±00.03 86.36±01.23
Kotlin 3.926±0.328 38.00±00.26 + 77.66±00.46 77.64±07.02
V/gcc 4.673±0.052 1.97±00.04 + 68.84±00.00 103.90±02.28
Python/pypy 6.511±0.192 64.35±00.14 + 69.21±00.02 114.96±05.01
C#/.NET Core 6.849±0.050 34.01±00.08 + 69.11±00.00 157.79±03.48
C#/Mono 11.242±0.494 20.21±00.08 + 69.07±00.01 216.89±23.82
Ruby/truffleruby 37.614±1.111 527.38±00.25 + 480.37±03.33 774.93±65.86
Ruby/truffleruby (--jvm) 71.812±1.623 605.31±08.13 + 318.65±11.88 1915.91±107.73
Ruby (--jit) 213.953±4.452 15.11±00.02 + 68.87±00.03 4556.86±112.03
Ruby 219.297±6.115 15.11±00.01 + 68.64±00.00 4348.97±369.08
Python 248.815±6.912 10.54±00.03 + 68.58±00.00 5171.34±260.75
Tcl 348.512±8.997 7.16±00.06 + 400.44±00.06 7136.61±492.96
Perl 396.918±3.594 9.00±00.04 + 599.61±00.05 9398.78±528.90
Ruby/jruby 500.782±14.495 270.32±06.16 + 670.69±06.01 10825.70±291.87

Tests Execution

Environment

CPU: Intel(R) Core(TM) i7-10710U

Base Docker image: Debian GNU/Linux bullseye/sid

Language Version
.NET Core 5.0.101
C#/.NET Core 3.8.0-5.20570.14 (6559f38c)
C#/Mono 6.12.0.107
C/clang 11.0.0
C/gcc 10.2.1
Chez Scheme 9.5.4
Clojure "1.10.1"
Crystal 0.35.1
D/dmd v2.095.0
D/gdc 10.2.1
D/ldc2 1.24.0
Elixir 1.10.3
F#/.NET Core 11.0.0.0 for F# 5.0
Go go1.15.6
Go/gccgo 10.2.1
Haskell 8.10.3
Java 15.0.1
Julia v"1.5.3"
Kotlin 1.4.21
Lua Lua 5.4
Lua/luajit LuaJIT 2.1.0-beta3
MLton 20201002
Nim 1.4.2
Node.js v15.5.1
OCaml 4.11.1
PHP 7.4.11
Perl v5.32.0
Python 3.9.1
Python/pypy 7.3.3-beta0 for Python 3.7.9
Racket "7.9"
Ruby 3.0.0p0
Ruby/jruby 9.2.14.0
Ruby/truffleruby 20.3.0
Rust 1.49.0
Scala 2.13.4
Swift swift-5.3.2-RELEASE
Tcl 8.6
V 0.2
Vala 0.48.12

Using Docker

Build the image:

$ docker build docker/ -t benchmarks

Run the image:

$ docker run -it --rm -v $(pwd):/src benchmarks <cmd>

where is:

  • versions (print installed language versions);
  • shell (start the shell);
  • brainfuck bench (build and run Brainfuck bench.b benchmarks);
  • brainfuck mandel (build and run Brainfuck mandel.b benchmarks);
  • base64 (build and run Base64 benchmarks);
  • json (build and run Json benchmarks);
  • matmul (build and run Matmul benchmarks);

Please note that the actual measurements provided in the project are taken semi-manually (via shell) as the full update takes days and could have occassional issues in Docker.

There is a Makefile that could be used to simlify Docker usage:

  • make build (build the image);
  • make versions (run the image with the versions command);
  • make shell (run the image with the `shell' command);
  • make toc (utility rule to update ToC in this README, requires git-markdown-toc available in PATH).

Please note that the make shell rule requires cpupower utility installed that is invoked with sudo to set cpufreq's performance governon (it runs the CPU at the maximum frequence to eliminate throttling issues).

Manual Execution

Makefiles contain recipes for building and executing tests with the proper dependencies. Please use make run (and make run2 where applicable). The measurements are taken using analyze.rb script:

$ cd <test suite>
$ ../analyze.rb make run
$ ../analyze.rb make run[<single test>]

Please note that the measurements could take hours. It uses 10 iterations by default, but it could be changed using ATTEMPTS environment variable:

$ ATTEMPTS=1 ../analyze.rb make run

Prerequisites

Please use Dockerfile as a reference regarding which packages and tools are required.

For all (optional):

  • Powercap for reading energy counters in Linux (Debian package powercap-utils).

For Python:

  • NumPy for matmul tests (Debian package python3-numpy).
  • UltraJSON for JSON tests (Debian package python3-ujson).

For C++:

  • Boost for JSON tests (Debian package libboost-dev).
  • JSON-C for JSON tests (Debian package libjson-c-dev).

For Rust:

  • libjq for jq test (Debian packages libjq-dev, libonig-dev and environment variable JQ_LIB_DIR=/usr/lib/x86_64-linux-gnu/).

For Java, Scala:

  • Coursier for downloading Maven artifacts.

For Lua:

  • LuaRocks for installing dependencies (Debian package luarocks).

For Haskell:

  • network for TCP connectivity between the tests and the test runner.
  • raw-strings-qq for raw string literals used in tests.

For Perl:

  • cpanminus for installing modules from CPAN (Debian package cpanminus).

Contribution

Please follow the criteria specified in the overview. Besides that please ensure that the communication protocol between a test and the test runner is satisfied:

  • The test runner listens on localhost:9001;
  • All messages are sent using TCP sockets closed immediately after the message has been sent;
  • There are two messages sent from a test (it establishes the measurement boundary):
    1. The beginning message having the format name of the test/tprocess ID (the process ID is used to measure the memory consumption). Please note that the name of the test couldn't use Tab character as it's a delimiter;
    2. The end message with any content (mostly it's "stop" for consistency).
  • The test runner could be unavailable (if the test is launched as is) and the test should gracefully handle it.

Makefile guide

Binary executables

If the test is compiled into a single binary, then two sections of the Makefile require changes:

  • append a new target (the final binary location) into executables variable;
  • append the proper target rule.

Compiled artifacts

If the test is compiled, but can't be executed directly as a binary, then three sections of the Makefile require changes:

  • append a new target (the final artifact location) into artifacts variable;
  • append the proper target rule to compile the test;
  • append run[<target_artifact>] rule to run the test.

Scripting language

If the test doesn't require compilation, then two sections of the Makefile requires changes:

  • append run[<script_file>] into all_runners variable;
  • append run[<script_file>] rule to run the test.