Small project to compare power usage of different stacks for a small rest api endpoint
In 2017, a group of researchers from Portugal released a paper comparing commonly used programming languages by their electricity usage while running a benchmark called 'The computer language benchmarks game'. The paper is great and I encourage everyone (interested in green software) to read it, as it tries to answer many interesting questions, such as:
- faster programs are always the most energy efficient?
- how does memory usage relate to energy consumption?
But he main question it tried to answer is: what is the most 'green' programming language?
I won't delve into the results (which are very insteresting and cathegorize the programming languages in combination of characteristics where they excel at) as the paper is easy to find and more importantly easy to read. However, I do have some minor issues with it:
- the workload is not, in my opinion, a good representation of 99% of application written nowadays and they mostly favor native compilation
- the way it was measured (power usage for a single shot execution) favors 'startup' speed, but most web application nowadays are long lived, so it would make sense to measure power usage after warmup for JIT compiled languages (java, javascript/typescript, jruby)
- the quality of the softwares used varied greatly, likely giving advantage to more popular languages
Most of the time, current web application or api based application are not going to be limited by CPU, and will most likely be waiting for IO. Therefore, I'm proposing a different benchmark: a rest based API backed by a PosgreSQL database and serializing data using JSON, a very typical stack.
The api has only a few endpoints that should be mostly straightforward and the open API yaml can be found here and the json version can be found here
I will publish them on github pages but very friefly:
/products
/products/{id}
/order/new (there's a json payload with the items)
/order/{id}
An implementation of this API should follow very basic rules:
- no caching of query results across requests
- stateless
- it doesn't need to do anything special, but it should compute the total price properly (which means making a number of queries when a new order is posted)
A database dump can be found here. I personally use docker and start it as following:
docker run --rm -e POSTGRES_USER=myself -e POSTGRES_PASSWORD=mysafepassword -e POSTGRES_DB=mydatabase -p 5432:5432 -v $PWD/scripts:/docker-entrypoint-initdb.d/ --name pgsql postgres:14.7
from the folder containing the scripts/ directory. Keep in mind to take notice of the user/password/database/port combinations you used to configure your application
The power usage will be measured using intel RAPL (same as in the paper) and exported using Scaphandre and for each implementation these will be the steps:
- start scaphandre on the test server
- disable CPU scale-down on the test server (modern CPU's will slow down when there's no load to not waste so much energy)
- take a baseline power measure for roughly 1 hour (the server will be mostly idle but I imagine it will still draw some power). This will be deducted from the power usage when running the application
- start a fresh database (not in the same server as the application)
- run benchmark (the clients will be run from the same machine as the database)
- stop everything and collect measures (subtracting the power usage from the baseline measure before)
Each 'story' will be made of four http requests:
- GET to /products
- GET to 5 random products
- POST to /order/new with the 5 products
- GET to /order/{id}
I've done two tests:
- [Fixed Request Rate for a set duration] (Test1.md)
- [Non stop requests for a set duration] (Test2.md)
There are a couple more tests that I want to make, but not sure if I will get to:
- hot application, maximum load: X threads making requests non-stop
- GraalVM native compilation has been getting a lot of attention lately, but native compilation takes a long time and uses a lot of CPU. I would like to know how many requests it takes for the Hotspot (or graalvm) JIT to be more energy efficient taking the native compilation into account
First implementation and the one I used to generate the openapi and database schema (that's the reason the constraints have such funky names, as they were generated by hibernate). To start it, from the command line, run
./gradlew quarkusDev
Running on a dev environment like above will create a fresh PostgreSQL database (using test container), create the schema and start serving on port 8080. To populate the database:
curl -XPOST http://localhost:8080/products/random
To access the openapi ui, point your browser at swagger ui here
To run in 'production', pack your application ./gradlew build, create a .env file with details on the database
you're using:
quarkus.datasource.username = myself
quarkus.datasource.password = mysafepassword
quarkus.datasource.jdbc.url = jdbc:postgresql://localhost:5432/mydatabase
and run with java -jar ./build/quarkus-app/quarkus-run.jar it should start listening on port 8080 (but the swagger
ui will not be available)
I wanted to use the most representative stack for each programming language. But since I couldn't find out what is scala's so I opted for using scala's most distinctive stack: cats + circe + http4s + doobie. Unfortunatelly, due to my own inexperience with this stack, this implementation is not good (and I'm accepting contributions of course).
While Kotlin can be used with most modern java frameworks, here again I opted for a 'unique' stack using Ktor. I didn't know at all this stack, but it was not difficult to learn (provided you know a bit of the language).
Thanks to Bruno Ortiz for this implementation. Its a very straightforward python + django, not much to say.
Hand built python version that does (almost) everything without any framework, courtesy of Dr. Jude Dekeyser. There are 4 variations, including non blocking ones.
Also not much to say, ruby on rails is a very powerful and easy to use framework.
NodeJS straightforward implementation by Kuroki
Golang straightforward implementation by Kuroki
Rust straightforward implementation by Zezinho
TBD