This is a WIP and is by no means complete. The goal of this project is to look at rack-based ruby frameworks, all who use Puma as a web server and look at load in terms of concurrency and average response time. The load profiling is done with Apache AB 2.4.
Benchmark constants:
-
ab
is run with a concurrency of100
and a total request size of10,000
. -
Puma
is run in clustered mode with a worker size of 2 and thread size of 16 min and 16 max. The worker size is based on the number of available cores you have so feel free to adjust. I left the thread size fixed in order to prevent any Puma optimizations that I might not be aware of. We usepreload
in order to take advantage of Copy on Write. -
The app must respond with a simple JSON payload of
{ "test": 123 }
when a GET request is made to the root path (ie http://127.0.0.1/)
The benchmark script is providing very strange results, I'm still trying to understand why.
Based on the benchmark script having wonky results, I suggest running this yourself as follows:
cd cuba
bundle exec puma -t 16:16 -w 2 --preload
# open new terminal window
ab -n 10000 -c 100 http://127.0.0.1:9292/
Requests per second: 4058.79 [#/sec] (mean)
Time per request: 24.638 [ms] (mean)
Time per request: 0.246 [ms] (mean, across all concurrent requests)
Requests per second: 3382.01 [#/sec] (mean)
Time per request: 29.568 [ms] (mean)
Time per request: 0.296 [ms] (mean, across all concurrent requests)
Requests per second: 2523.10 [#/sec] (mean)
Time per request: 39.634 [ms] (mean)
Time per request: 0.396 [ms] (mean, across all concurrent requests)
Requests per second: 4468.30 [#/sec] (mean)
Time per request: 22.380 [ms] (mean)
Time per request: 0.224 [ms] (mean, across all concurrent requests)
Take the results above with a grain of salt. I'm still tweaking the set up of these frameworks to hopefully compare apples to apples. I'm very surprised Sinatra is running at 60% of Cuba or Hanami and I'm looking into other configuration details to speed up Sinatra if possible.
- I've started to run ruby-prof to identify bottlenecks in each framework and hopefully come up with some optimization strategies for each.
- Fix benchmark script
- Profiling Sinatra in the script is producing strange results, in fact the server crashes after ~6.4k requests.
- Auto adjust Puma's cluster size based on machine's cores
- Produce a better overview than ab's default output
- Call out versions of everything
- There are a few frameworks I left out such as NYNY, Nancy, etc. Those looked to be stale and not receiving updates so I punted on them.
- Rack 2.0 is available in Sinatra and Rails.