/benchmarking

Instructions for benchmarking Ewasm precompiles. (And results.)

Primary LanguagePythonApache License 2.0Apache-2.0

Benchmarks

This repository contains instructions for benchmarking evm implementations, ewasm contracts and standalone wasm modules. Directory descriptions follow.

evm/            - contains benchmarks for different evm implementations (geth, parity, cita-vm, evmone)
ewasm/          - contains benchmarks and tests for ewasm contracts in ewasm engines.
wasm/           - contains benchmarks for wasm modules in standalone wasm engines.
wasm-engines/   - contains benchmarks for wasm modules comparing wasm engines.

EVM

Directory /evm contains a list of the current benchmarked evm implementations:

evm/
  cita-vm/
  evmone/
  geth/
  parity/

Build each one of the evm implementations:

$ make build_evm_engines

Run EVM benchmarks:

$ make benchmark_evm_engines

The previous command will create a new directory evmraceresults, containing the following files:

evm_benchmarks.csv         - consolidated benchmarks
evm_benchmarks_evmone.csv  - evmone benchmarks
evm_benchmarks_parity.csv  - parity benchmarks
evm_benchmarks_geth.csv    - geth benchmarks
evm_benchmarks_cita-vm.csv - cita vm benchmarks

Run precompiles benchmarks:

  • Geth:
$ cd evm
$ ./scripts/run_precompiles_bench.py geth
  • Parity
$ cd evm
$ ./scripts/run_precompiles_bench.py parity

Wasm Engines Benchmarks

Build the docker image:

$ make build_wasm_engines

Build the wasm binaries and execute benchmarks:

$ cd wasm-engines && ./run_benchmarks.sh

Scout Benchmarks

Build the docker image:

$ make build_scout_engines

Run benchmarks

make benchmark_scout_engines

Generate charts

Install python deps for plotting benchmark graphs:

$ pip install -r requirements.txt
$ make generate_charts

EVM engine versions

  • cita-vm: v0.1.3 (May 2019)
  • evmone: v0.5.0 (Jun 2020)
  • geth: v1.9.14 (+ go 1.11) (May 2020)
  • parity/openethereum: v2.5.1 (May 2019)

WebAssembly engine versions

  • asmble: v0.4.2-fl (+ openjdk-8-jre) (Oct 2018)
  • fizzy: v0.3.0 (Jul 2020)
  • life: d05763d1 (Feb 2019)
  • ssvm: v0.6.0 (Jun 2020)
  • v8 (node.js): v12.18.2 (Jun 2020)
  • modified/optimised wabt: 04fe0c41 (closest to v1.0.9) (Mar 2019)
  • "vanilla" wabt: v1.0.17 (Jun 2020)
  • wagon: v0.4.0 (Mar 2019)
  • wamr: WAMR-02-18-2020 (Feb 2020)
  • wasm3: v0.4.7 (Apr 2020)
  • wasmi: v0.4.4 (Mar 2019)
  • wasmtime: fb7c1b77 (closest to v0.31.0) (Feb 2019)
  • wavm: d3607084 (closest to nightly-2019-08-28) (Aug 2019)