BalancerBattle was a load test against load balancers / proxies that support WebSockets. These pieces of technology are vital if you want to scale to multiple servers or processes.
The following technologies were tested:
- http-proxy, version: 0.10.0
- nginx, version: 1.3.15 (development release)
- HAProxy, version: 1.5-dev18 (development release)
- Nothing, just the plain echo server that was used as a control test.
3 different, separate servers were used for testing. All these servers are hosted at joyent.
- Proxy, a 512mb Ubuntu server. This is the server were all the proxy servers are installed. image: sdc:jpc:ubuntu-12.04:2.4.0
- WebSocketServer, a 512mb Node.js smart machine that ran our WebSocket echo
server. The server is written in Node.js and spread across multiple cores
using the
clustermodule. image: sdc:sdc:nodejs:1.4.0 - Thor, another 512mb Node.js smart machine with the same specs as above. This was the server were we generated the load from. Thor is a WebSocket load generation tool which we've developed. It's released as open source and available at http://github.com/observing/thor
The Proxy server was just a clean, bare bones Ubuntu 12.04 server. These are the steps that were taken to configure and install all the dependencies. To ensure that everything is up to date we have to run.
apt-get upgrade
The following dependencies were installed on the system:
gitfor access to the github repositories.build-essentialfor compiling the proxies for source, most of the proxies recently got support for WebSockets or HTTPS.libssl-devNeeded for HTTPS support.libev-devLibev required for stud, stud is awesomesss.
apt-get install git build-essential libssl-dev libev-dev
Node.js is required for the http-proxy. While it runs on the latest Node.js
version for these tests were executed under 0.8.19 to ensure compatibility of
all dependencies. It was installed through github.
git clone git://github.com/joyent/node.git
cd node
git checkout v0.8.19
./configure
make
make install
This also installed the npm binary on the system so we can install the
dependencies of this project. Run npm install . in the root of this repository
and the http-proxy and all it's dependencies are installed automatically.
Nginx is already a widely deployed server. It supports proxing of to different back end servers but it did not support WebSockets. This got recently added in to the development branch of Nginx. There for we installed the latest development version and compiled from source:
wget http://nginx.org/download/nginx-1.3.15.tar.gz
tar xzvf nginx-1.3.15.tar.gz
cd nginx-1.3.15
./configure --with-http_spdy_module --with-http_ssl_module \
--pid-path=/var/run/nginx.pid --conf-path=/etc/nginx/nginx.conf \
--sbin-path=/usr/local/sbin --http-log-path=/var/log/nginx/access.log \
--error-log-path=/var/log/nginx/error.log --without-http_rewrite_module
As you can from the options above we've included SSL, SPDY and configured some other settings. This yielded the following configuration summary:
Configuration summary
+ PCRE library is not used
+ using system OpenSSL library
+ md5: using OpenSSL library
+ sha1: using OpenSSL library
+ using system zlib library
nginx path prefix: "/usr/local/nginx"
nginx binary file: "/usr/local/sbin"
nginx configuration prefix: "/etc/nginx"
nginx configuration file: "/etc/nginx/nginx.conf"
nginx pid file: "/var/run/nginx.pid"
nginx error log file: "/var/log/nginx/error.log"
nginx http access log file: "/var/log/nginx/access.log"
nginx http client request body temporary files: "client_body_temp"
nginx http proxy temporary files: "proxy_temp"
nginx http fastcgi temporary files: "fastcgi_temp"
nginx http uwsgi temporary files: "uwsgi_temp"
nginx http scgi temporary files: "scgi_temp"
After this it's just a simple make away:
make
make install
HAProxy was already able to proxy WebSockets in tcp mode but it's now also
possible to do so in http mode. HAProxy also got support for HTTPS
termination. So again, we need to install the development branch.
wget http://haproxy.1wt.eu/download/1.5/src/devel/haproxy-1.5-dev18.tar.gz
tar xzvf haproxy-1.5-dev18.tar.gz
cd haproxy-1.5-dev18
make TARGET=linux26 USE_OPENSSL=1
make install
While HAProxy is capable of terminating SSL it's common practise to have stud in front of HAProxy for SSL offloading. So this is something we want to test as well.
git clone git://github.com/bumptech/stud.git
cd stud
make
make install
Now that everything is installed we need to install the configuration files. For
Nginx you can copy & paste the nginx.conf from the root of this repository to
/etct/nginx/nginx.conf. All the other proxies can be configured on the fly.
After all the proxies are installed we need to do some socket tuning. This information was generously stolen from the internets:
vim /etc/sysctl.conf
And set the following values.
# General gigabit tuning:
net.core.somaxconn = 16384
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_syncookies = 1
# this gives the kernel more memory for tcp
# which you need with many (100k+) open socket connections
net.ipv4.tcp_mem = 50576 64768 98152
net.core.netdev_max_backlog = 2500
There are 2 different tests executed:
- Load testing the proxies without SSL. This will purely test the performance of WebSocket proxing.
- Load testing the proxies with SSL. Nobody should be running unsecured WebSockets as they have really bad connectivity in browsers. But this adds overhead of SSL termination to the proxy.
In addition to different tests we're also testing the different amount of connections:
- 2k
- 5k
- 10k
And for the equal results:
- 20k
- 30k
Before each test all WebSocketServer is reset and the Proxy re-initiated. Thor
will hammer all the Proxy server with x amount of connection with a
concurrency of 100. For each established connection one single UTF-8 message
is send and received. After the message is received the connection is closed.
stud --config stud.conf
haproxy -f ./haproxy.cfg
nginx
FLAVOR=http node http-proxy.js
FLAVOR=http node index.js
The http-proxy lives up to it's name, it proxies requests and does it quite
fast. But as it's build on top of Node.js it quite heavy on the memory. Just a
simple node process starts with a 12MB of memory. For the 10K requests it took
around 70mb of memory. The overhead was of the HTTP proxy was about 5 seconds
if you compare it to control test. The HTTPS test was the slowest of all, but
that was expected as Node.js sucks hairy monkey balls in SSL. Not to mention
that will put your event loop to a grinding halt when it's under severe stress.
There is a pull request
for the http-proxy that will drasticly reduce memory consumption. I've
manually applied the patch and saw the memory consumption get cut in half. It's
still using a lot more compared to nginx after this patch that can easily be
explained because they are all build in pure c.
I had high hopes for Nginx and it did not let me down. It had a peak memory of
10MB and it was really fast. The first time I tested Nginx, it had a horrible
performance. Node was even faster in SSL then Nginx, I felt like failure, I
genuinely sucked a configuring Nginx. But after some quick tips from some
friends it was actually a one line change in the config. I had the wrong
ciphers configured. After some quick tweaking and a confirmation using
openssl s_client -connect server:ip it was all good and used RC4 by default
which is really fast.
Up next was HAproxy, it has the same performance profile as NGINX, but lower on
the memory it only required 7MB of memory. The biggest difference was when we
tested with HTTPS. It's was really slow and no where near the performance of
Nginx. Hopefully this will be resolved as it's a development branch we are
testing. Made the same mistake as I did with Nginx, configured the wrong
ciphers which was kindly pointed out on
HackerNews. In addition to
testing HTTPS we also put stud in front of it to see what kind of performance
it would yield.
http-proxy it's a great flexible proxy, really easy to extend and build up on.
If you deploy this in production I advice to run stud in front of it to take
care of the SSL offloading.
nginx and haproxy were really close, it's almost not significant enough to
say that one is faster or better then the other. But if you look at it from an
operations stand point. It's easier to deploy and manage a single nginx server
instead of stud and haproxy
| Proxy | Connections | Handshaken (mean) | Latency (mean) | Total |
|---|---|---|---|---|
| http-proxy | 10k | 293 ms | 44 ms | 30168 ms |
| nginx | 10k | 252 ms | 16 ms | 28433 ms |
| haproxy | 10k | 209 ms | 18 ms | 26974 ms |
| control | 10k | 189 ms | 16 ms | 25310 ms |
Winner: Both Nginx and HAProxy are really fast and close to each other.
| Proxy | Connections | Handshaken (mean) | Latency (mean) | Total |
|---|---|---|---|---|
| http-proxy | 10k | 679 ms | 62 ms | 68670 ms |
| nginx | 10k | 470 ms | 30 ms | 50180 ms |
| haproxy | 10k | 464 ms | 25 ms | 50058 ms |
| haproxy + stud | 10k | 492 ms | 42 ms | 52403 ms |
| control | 10k | 703 ms | 65 ms | 71500 ms |
Winner: Both Nginx and HAProxy are really fast and close to each other.
All test results are available at:
https://github.com/observing/balancerbattle/tree/master/results
All the configuration are in the repository, I'm more then happy to see if we can squeeze more performance out of the servers.