kolotaev/servo-bench

Benchmark of servers and web-frameworks for small machines

Benchmark of various web-frameworks

Tests for minimal machines with limited resources (1 vCPU, 1 Gb RAM).

Note that all tests are performed on a dedicated vCPU. Results may (and will) vary for basic VPS plans that use shared CPU resources.

• Benchmark of various web-frameworks
  • Tests
  • Frameworks
    • Go. Echo.
    • JS. Express.
    • Crystal. Kemal.
    • Python. Flask. Gevent
    • Python. Aiohttp. UVloop.
    • Clojure. Ring. Jetty.
    • Clojure. Http-kit.
    • Clojure. Undertow.
    • Scala. Play. Akka.
    • Elixir. Phoenix. OTP.
    • Elixir. Plug. OTP.
    • Ruby. Synchrony.
    • Ruby. Sinatra. Puma.
    • Lua. Openresty. Nginx.
    • Java. Vertx.
    • Haskell. Hasura.
  • Mule script

Tests

• Simple random tiny JSON response.

  wrk -t12 -c400 -d240s http://vagrant-machine:8080/json --timeout 10s

  Detailed results can be found here.

  

• Postgres DB call that does sleep random(2 sec) mimicking random long queries and random js response generated 100 times to simulate post-processing. DB pool size is 400.

  wrk -t12 -c400 -d240s http://vagrant-machine:8080/db --timeout 10s

  Detailed results can be found here.

  

• Postgres DB call that does SELECT count(*) FROM pg_catalog.pg_user mimicking fast queries and random js response generated 100 times to simulate post-processing. DB pool size is 400.

  wrk -t12 -c400 -d240s http://vagrant-machine:8080/db --timeout 10s

  Detailed results can be found here.

  

Frameworks

Every framework section has 2 (or 3) words: Language. Framework name. (Web server name or networking library).

Go. Echo.

• Echo

Logging disabled.

JS. Express.

• NodeJS site
• Express site

Crystal. Kemal.

• Crystal lang
• Kemal web framework

Python. Flask. Gevent

• Flask
• Gunicorn
• Gevent

With setup: 'Gunicorn as a web-server. 4 workers. Worker type - async: eventlet' server gives only 3-4 req/s which is very low.

Tuned bare gevent is much better.

Python. Aiohttp. UVloop.

• Aiohttp
• uvloop
• acyncpg

Strange, but UVloop gives no benefits for db/ endpoint.

Clojure. Ring. Jetty.

• Ring
• Jetty

Jetty with all default settings.

Clojure. Http-kit.

• Http-kit
• Postgres.async

Event-driven, high-performance clojure server/client library. With async PG driver.

Clojure. Undertow.

• Unertow
• Undertow-ring

Scala. Play. Akka.

• Play
• Akka

Elixir. Phoenix. OTP.

• Elixir
• Phoenix
• Erlang & OTP

Elixir. Plug. OTP.

• Elixir
• Plug
• Erlang & OTP

Ruby. Synchrony.

• Ruby
• Synchrony

Ruby. Sinatra. Puma.

• Ruby
• Sinatra
• Puma

Lua. Openresty. Nginx.

• Luajit
• Openresty
• Nginx

Java. Vertx.

• Vertx

For DB we use async vert.x postgres driver

Haskell. Hasura.

• Hasura

Doesn't currently participate in the race, but you can see the results in its directory. It's quite promising.

Sleep Echo server

In order to test remote HTTP requests performance we spin up a simple Golang echo-server that sleeps for a specified amount of time before responding back.

Build and run it with a standard mule script command prior to tests.

Its single endpoint is: http://127.0.0.1:9000/sleep/${sleep_second_you_want}

Mule script

mule is a script to help you build and run specific framework in a docker-container.

Script can be run in 2 possible ways:

# 1. From repo's root directory: you need to specify directory of the framework in -d option.
./mule.sh -d go-echo

# 2. From specific framework's directory: no -d option is needed.
cd go-echo
../mule.sh

Available actions of using mule (assuming the 2nd way of running):

# Read help for `mule` script
../mule.sh -h

# Build all images
./mule.sh -x

# Build/rebuild image
../mule.sh -b

# Run container
../mule.sh -r

# Run container with attached TTY (to see its output)
../mule.sh -ra

# Kill running container
../mule.sh -k

# Build image, kill container if running, run new w/ attached TTY
../mule.sh -brak

# Run with specific options (loop-count, etc.)
../mule.sh -ra -l 1000