/chainhammer

fire many transactions at Ethereum node, then produce diagrams of TPS, blocktime, gasUsed and gasLimit, and blocksize.

Primary LanguagePythonOtherNOASSERTION

NEWS 2019-Feb-23 - video!
Explaining software v55 video released on youtube:

--> watch

chainhammer-logo.png


chainhammer v59

TPS measurements of parity aura, geth clique, quorum, tobalaba, etc. It should work with any Ethereum type chain; we focused on PoA consensus.

instructions

video

The brand new release v55 is now presented & explained in a useful video on youtube.

folders

  • hammer/ - submits many transactions, while watching the recent blocks
  • reader/ - reads blocks; visualizes TPS, blocktime, gas, bytes - see reader/README.md
  • docs/ - see esp. reproduce.md, cloud.md, FAQ.md, new: azure.md
  • results/ - for each client one markdown file; results/runs/ - auto-generated pages
  • logs/ - check this first if problems
  • networks/ - network starters & external repos via install script, see below
  • scripts/ - installers and other iseful bash scripts
  • env/ - Python virtualenv, created via install script, see below
  • tests/ - start whole integration test suite via ./pytest.sh

chronology

See the results/ folder:

  1. log.md: initial steps; also tried Quorum's private transactions
  2. quorum.md: raft consensus, quorum is a geth fork
  3. tobalaba.md: parity fork of EnergyWebFoundation
  4. quorum-IBFT.md: IstanbulBFT, 2nd consensus algo in quorum
  5. geth.md: geth clique PoA algorithm
  6. parity.md: parity aura PoA algorithm, many attempts to accelerate
  7. eos.md: not begun yet
  8. substrate.md: not begun yet

results summary

Outdated table in which I had run each of the experiments manually in autumn 2018; soon re-done completely, using the below automation. So please contact me now, if you know how to accelerate any of these clients:

hardware node type #nodes config peak TPS_av final TPS_av
t2.micro parity aura 4 (D) 45.5 44.3
t2.large parity aura 4 (D) 53.5 52.9
t2.xlarge parity aura 4 (J) 57.1 56.4
t2.2xlarge parity aura 4 (D) 57.6 57.6
t2.micro parity instantseal 1 (G) 42.3 42.3
t2.xlarge parity instantseal 1 (J) 48.1 48.1
t2.2xlarge geth clique 3+1 +2 (B) 421.6 400.0
t2.xlarge geth clique 3+1 +2 (B) 386.1 321.5
t2.xlarge geth clique 3+1 (K) 372.6 325.3
t2.large geth clique 3+1 +2 (B) 170.7 169.4
t2.small geth clique 3+1 +2 (B) 96.8 96.5
t2.micro geth clique 3+1 (H) 124.3 122.4
t2.micro SWAP quorum crux IBFT 4 (I) SWAP! 98.1 98.1
t2.micro quorum crux IBFT 4 (F) lack of RAM
t2.large quorum crux IBFT 4 (F) 207.7 199.9
t2.xlarge quorum crux IBFT 4 (F) 439.5 395.7
t2.xlarge quorum crux IBFT 4 (L) 389.1 338.9
t2.2xlarge quorum crux IBFT 4 (F) 435.4 423.1
c5.4xlarge quorum crux IBFT 4 (F) test_getNearestEntry() 536.4 524.3

Reproduce these results easily; for the config column also see there. Quickest reproduction with my Amazon AMI readymade image. And see that bottom of parity.md and geth.md and quorum-IBFT.md for the latest runs, issues, and additional details.

faster wider more

  • how I initially got this faster, on Quorum, step by step, please do read the 1st logbook log.md
  • then I improved per client, see each in #chronology above
  • (possible TODOs - any other ideas?)

but not much more needed = the current version is already fully automated. Use it! May it help you to improve the speed of your Ethereum client!

you

Add yourself to other-projects.md using chainhammer, or projects which are similar to this.

(Especially if you work in one of the dev teams, you know your client code best - ) please try to improve the above results, e.g. by varying the CLI arguments with which the nodes are started; I don't see that as my job, you will be much more successful with that.

See parity PE#9393, parity SE#58521, geth GE#17447, quorum Q#479.

Please report back when you have done other / new measurements.

install and run

All this is developed and much tested on Debian, locally and in the AWS cloud. New: Ubuntu now also supported, see below.

quickstart

N.B.: Better do this on a disposable cloud, or virtualbox machine; because the installation makes lasting changes and needs sudo!

After unpacking a ZIP of the downloaded repo, or by

git clone https://github.com/drandreaskrueger/chainhammer drandreaskrueger_chainhammer
ln -s drandreaskrueger_chainhammer CH
cd CH

you now only need these two lines to prepare and run the 1st experiment!

scripts/install.sh
CH_TXS=1000 CH_THREADING="sequential" ./run.sh $HOSTNAME-TestRPC testrpc

You will then have a diagram, and a HTML and MD page about this run!

(on Ubuntu instead: scripts/install.sh docker ubuntu )

activate docker

Better now logout & login, or close the terminal, and open a new terminal, because the above scripts/install.sh might have enabled docker for the the first time for this user. Then:

All supported clients in one go:

For the full integration test, run each client for a short moment:

export CH_MACHINE=yourChoice
./run-all_small.sh

For detailed instructions, please see docs/, esp. reproduce.md, and for troubleshooting FAQ.md and github issues.

benchmarking a remote node

Chainhammer can now be stripped down to its pure benchmarking abilities, i.e. without the installation of docker and without the three local network starters (parity-deploy, geth-dev, quorum-crux). It was successfully used to benchmark the Microsoft Azure blockchain-as-a-service product. The essential difference is to start the installation with the switch nodocker:

scripts/install.sh nodocker

So, if you just want to benchmark your existing Ethereum node or network, have a look at the manual docs/azure.md .

unittests

./pytest.sh

enables the virtualenv, then starts a testrpc-py Ethereum simulator on http://localhost:8545 in the background, logging into tests/logs/; then runs ./deploy.py andtests; and finally runs all the unittests, also logging into tests/logs/.

(Instead of testrpc-py) if you want to run tests with another node, just start that; and run pytest manually:

source env/bin/activate
py.test -v --cov

There were 98 tests on January 23rd, all 98 PASSED (see this logfile --> cat tests/logs/*.ansi because colors) on these different Ethereum providers:

  • testrpc instantseal (testrpc-py) 13 seconds
  • geth Clique (geth-dev) 63 seconds
  • quorum IBFT (blk-io/crux) 59 seconds
  • parity instantseal (parity-deploy) 8 seconds
  • parity aura (parity-deploy) 72 seconds

credits

Please credit this as:

benchmarking scripts "chainhammer"
maintainer: Dr Andreas Krueger 2018-2020
https://github.com/drandreaskrueger/chainhammer

Consider to submit your improvements & usage as pull request. Thanks.

development was supported by

v01-v35 financed by Electron.org.uk 2018
v40-v55 financed by Web3Foundation 2018-2019
v58-v59 financed by Microsoft Azure 2019

logo

Thank you very much!

short summary

The open source tools 'chainhammer' submits a high load of smart contract transactions to an Ethereum based blockchain, then 'chainreader' reads the whole chain, and produces diagrams of TPS, blocktime, gasUsed and gasLimit, and the blocksize. https://github.com/drandreaskrueger/chainhammer




# The following diagrams are outdated! Just make your own, new ones, with:
CH_MACHINE=yourChoice ./run-all_large.sh

chainhammer: hammer --> reader --> diagrams

examples:

geth clique on AWS t2.xlarge

geth.md = geth (go ethereum client), "Clique" consensus.

50,000 transactions to an Amazon t2.xlarge machine.

Interesting artifact that after ~14k transactions, the speed drops considerably - but recovers again. Reported.

geth-clique-50kTx_t2xlarge_tps-bt-bs-gas_blks12-98.png
reader/img/geth-clique-50kTx_t2xlarge_tps-bt-bs-gas_blks12-98.png

quorum IBFT on AWS t2.xlarge

quorum-IBFT.md = Quorum (geth fork), IBFT consensus, 20 millions gasLimit, 1 second istanbul.blockperiod; 20000 transactions multi-threaded with 23 workers. Initial average >400 TPS then drops to below 300 TPS, see quorum issue)

quorum-crux-IBFT_t2xlarge_tps-bt-bs-gas_blks320-395.png

quorum raft

OLD RUN on a desktop machine.

quorum.md = Quorum (geth fork), raft consensus, 1000 transactions multi-threaded with 23 workers, average TPS around 160 TPS, and 20 raft blocks per second) reader/img/quorum_tps-bt-bs-gas_blks242-357.png

tobalaba

OLD RUN on a desktop machine.

tobalaba.md = Public "Tobalaba" chain of the EnergyWebFoundation (parity fork), PoA; 20k transactions; > 150 TPS if client is well-connected.

reader/img/tobalaba_tps-bt-bs-gas_blks5173630-5173671.png

parity aura v1.11.11 on AWS t2.xlarge

parity.md#run-18 = using parity-deploy.sh dockerized network of 4 local nodes with increased gasLimit, and 5 seconds blocktime; 20k transactions; ~ 60 TPS on an Amazon t2.xlarge machine.

N.B.: Could not work with parity v2 yet because of bugs PD#76 and PE#9582 --> everything still on parity v1.11.11

parity-v1.11.11-aura_t2xlarge_tps-bt-bs-gas_blks5-85.png
parity-v1.11.11-aura_t2xlarge_tps-bt-bs-gas_blks5-85.png

Calling all parity experts: How to improve these too slow TPS results?

See issue PE#9393, and the detailed log of what I've tried already, and the 2 shortest routes to reproducing the results: reproduce.md.

Thanks.