/WebFrameworkBenchmark

Benchmarking some web frameworks

Primary LanguageJava

Web Frameworks Benchmark

The idea behind this benchmark is to re-evaluate results presented by Techempower's Benchmark benchmarking best & promising open-source frameworks.

Why another benchmark?

I just found Techempower's Benchmark sources overcomplicated. Also wanted to test only framework overhead, that's why testing just trivial dynamic Hello World world output.

Just to not raise a controversy, I want also to emphasize this benchmark is simple & native and completely skips some unique features of some frameworks and languages, such as Erlang's natural clustering and hot-swap capabilities.

Results

Language Framework Req/sec1 MB/sec 99% ms2
Java Undertow 616 547 80.55 3.29
C Kore 3 572 782 104.33 3.99
C libmicrohttpd 533 626 69.72 1.28
Go fasthttp 485 185 67.56 5.26
C Onion 4 483 824 90.90 2.82
Java Netty 422 580 40.30 4.08
Nim m&s 6 AsyncHTTPServer 404 040 43.93 3.98
Native Nginx 5 381 368 43.26 24.24
Go net/http 270 253 34.28 2.52
Lua OpenResty 7 269 205 30.28 43.35
C++ Crow 256 552 31.32 12.28
Rust Iron 178 789 19.44 0.05 8
Erlang Cowboy 9 163 521 24.01 5.41
Node HTTP 112 086 13.79 11.98
Nim m&s10 AsyncHTTPServer 86 741 9.43 1.40
Nim m&s10 Jester 11 83 753 5.99 1.50
Ruby Puma 12 83 053 6.02 6.14
D ldc2 13 Vibe.d 0.7.26 79 602 13.28 46.41
D dmd 14 Vibe.d 0.7.26 76 839 12.75 103.05
Nim 15 AsyncHTTPServer 52 843 5.75 4.76
Nim 15 Jester 42 698 3.05 5.42

1 Ubuntu 14.04 LTS, Linux 3.16, Xeon E5-1650 @ 3.50GHz, 32 GB RAM
2 Latency distribution value at 99% in milliseconds (towards worst)

3 Core built without SSL via using make NOTLS=1.
4 Running hello example with static path.
5 Using Nginx echo module.
6 Nim using --gc:markandsweep, pre-forked processes using SO_REUSEPORT.
7 OpenResty is in fact Nginx with Lua module.
8 Rust Iron has some amazing super-stable latency in longer runs.
9 Cowboy requires some low level tweaking via sysctl, see and apply sysctl.conf.
10 Nim using --gc:markandsweep, single-thread only.
11 Jester is some higher-level web framework for Nim.
12 Using several Ruby instances with puma -w 12.
13 D language using LDC2 compiler v0.16.1 (LLVM 3.7.0).
14 D language using standard DMD compiler v2.069.1.
15 Nim using standard RC garbage collection, single-thread only.

NOTE: Detailed results can be found in results/.

Benchmarking details

Each web framework is expected to respond with Hello World content of type text/plain with minimum set of headers required by HTTP 1.1 specification:

$ curl -i localhost:8080
HTTP/1.1 200 OK
Connection: Keep-Alive
Content-Length: 11
Content-Type: text/plain
Date: Tue, 24 Nov 2015 17:32:30 GMT

Hello World

Some frameworks add extra headers such as Server or Expire by default, which are not required by HTTP 1.1 specification. If possible we use some settings or tweaks to remove them as more headers (so more data) will have negative impact on performance.

Effectively this benchmark tests solely the framework overhead itself. We are not testing any database access or JSON serialization performance. We are also avoiding some extra optimizations, such as caching response memory structures which could improve performance a bit, but contradicts dynamic behavior of tested frameworks.

Conclusions

As expected Java solution - Undertow is most optimized. 2nd & 3rd places are occupied by native C frameworks - Kore and libmicrohttpd.

Finally new fasthttp Go solution takes 4th place, being very close to the top 3. Go is very flexible little language, improving productivity and it is used already by many companies to deliver some heavy-load network services.

It has to be also observed that different frameworks generated different amount of data due different HTTP headers being used.

License

This benchmark is provided under MIT license:

Copyright (c) 2015 Adam Strzelecki

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.