benchmarks is not correct
skar404 opened this issue · 7 comments
skar404 commented
Hi, i run benchmarks and do not agree with your
Zero:
➜ wrk git:(master) ./wrk -d10s -t8 -c240 http://127.0.0.1:8000/order
Running 10s test @ http://127.0.0.1:8000/order
8 threads and 240 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 115.38ms 81.09ms 362.51ms 59.20%
Req/Sec 269.33 65.87 470.00 71.38%
21495 requests in 10.03s, 5.29MB read
Socket errors: connect 0, read 103, write 0, timeout 0
Requests/sec: 2143.93
Transfer/sec: 540.17KB
➜ wrk git:(master) ./wrk -d10s -t8 -c240 http://127.0.0.1:8000/order
Running 10s test @ http://127.0.0.1:8000/order
8 threads and 240 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 107.93ms 36.25ms 179.24ms 75.08%
Req/Sec 276.98 50.81 424.00 69.00%
22118 requests in 10.03s, 5.44MB read
Socket errors: connect 0, read 100, write 0, timeout 0
Requests/sec: 2204.56
Transfer/sec: 555.46KB
➜ wrk git:(master) ./wrk -d10s -t8 -c240 http://127.0.0.1:8000/order
Running 10s test @ http://127.0.0.1:8000/order
8 threads and 240 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 126.72ms 110.52ms 452.73ms 66.61%
Req/Sec 267.83 99.47 520.00 65.75%
21371 requests in 10.03s, 5.26MB read
Socket errors: connect 0, read 99, write 0, timeout 0
Requests/sec: 2130.75
Transfer/sec: 536.91KBSanic:
➜ wrk git:(master) ./wrk -d10s -t8 -c240 http://127.0.0.1:8000/order
Running 10s test @ http://127.0.0.1:8000/order
8 threads and 240 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 107.09ms 127.92ms 1.35s 95.22%
Req/Sec 350.38 75.99 524.00 78.76%
27779 requests in 10.07s, 4.87MB read
Socket errors: connect 0, read 150, write 0, timeout 0
Requests/sec: 2758.47
Transfer/sec: 495.66KB
➜ wrk git:(master) ./wrk -d10s -t8 -c240 http://127.0.0.1:8000/order
Running 10s test @ http://127.0.0.1:8000/order
8 threads and 240 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 86.60ms 29.02ms 332.78ms 74.45%
Req/Sec 347.26 65.74 520.00 74.25%
27842 requests in 10.08s, 4.89MB read
Socket errors: connect 0, read 149, write 0, timeout 0
Requests/sec: 2762.66
Transfer/sec: 496.42KB
➜ wrk git:(master) ./wrk -d10s -t8 -c240 http://127.0.0.1:8000/order
Running 10s test @ http://127.0.0.1:8000/order
8 threads and 240 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 84.77ms 30.17ms 506.58ms 76.15%
Req/Sec 355.97 59.02 554.00 77.88%
28529 requests in 10.07s, 5.01MB read
Socket errors: connect 0, read 149, write 1, timeout 0
Requests/sec: 2832.60
Transfer/sec: 508.98KBFastAPI:
➜ wrk git:(master) ./wrk -d10s -t8 -c240 http://127.0.0.1:8000/order
Running 10s test @ http://127.0.0.1:8000/order
8 threads and 240 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 184.36ms 206.19ms 1.23s 90.13%
Req/Sec 240.48 74.93 392.00 73.57%
17496 requests in 10.08s, 3.57MB read
Socket errors: connect 0, read 101, write 0, timeout 0
Requests/sec: 1736.22
Transfer/sec: 362.84KB
➜ wrk git:(master) ./wrk -d10s -t8 -c240 http://127.0.0.1:8000/order
Running 10s test @ http://127.0.0.1:8000/order
8 threads and 240 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 107.09ms 59.14ms 563.79ms 79.35%
Req/Sec 289.56 89.17 520.00 69.00%
23184 requests in 10.06s, 4.73MB read
Socket errors: connect 0, read 101, write 0, timeout 0
Requests/sec: 2303.92
Transfer/sec: 481.48KB
➜ wrk git:(master) ./wrk -d10s -t8 -c240 http://127.0.0.1:8000/order
Running 10s test @ http://127.0.0.1:8000/order
8 threads and 240 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 109.62ms 35.28ms 351.88ms 70.78%
Req/Sec 272.44 47.40 404.00 73.38%
21811 requests in 10.05s, 4.45MB read
Socket errors: connect 0, read 101, write 0, timeout 0
Requests/sec: 2169.43
Transfer/sec: 453.38KBaiohttp:
➜ wrk git:(master) ./wrk -d10s -t8 -c240 http://127.0.0.1:8000/order
Running 10s test @ http://127.0.0.1:8000/order
8 threads and 240 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 139.98ms 140.77ms 1.35s 94.33%
Req/Sec 261.75 58.19 431.00 76.31%
20519 requests in 10.07s, 4.93MB read
Socket errors: connect 0, read 101, write 0, timeout 0
Requests/sec: 2038.11
Transfer/sec: 501.57KB
➜ wrk git:(master) ./wrk -d10s -t8 -c240 http://127.0.0.1:8000/order
Running 10s test @ http://127.0.0.1:8000/order
8 threads and 240 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 117.28ms 88.65ms 939.02ms 95.08%
Req/Sec 282.75 63.12 424.00 78.28%
22440 requests in 10.06s, 5.39MB read
Socket errors: connect 0, read 101, write 0, timeout 0
Requests/sec: 2230.80
Transfer/sec: 549.03KB
➜ wrk git:(master) ./wrk -d10s -t8 -c240 http://127.0.0.1:8000/order
Running 10s test @ http://127.0.0.1:8000/order
8 threads and 240 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 108.08ms 58.23ms 803.84ms 88.32%
Req/Sec 287.95 61.60 421.00 74.87%
22947 requests in 10.05s, 5.51MB read
Socket errors: connect 0, read 96, write 0, timeout 0
Requests/sec: 2282.98
Transfer/sec: 561.84KBpip freeze:
aiofiles==0.7.0
aiohttp==3.7.4.post0
aioredis @ git+https://github.com/aio-libs/aioredis@ff5a8fe068ebda837d14c3b3777a6182e610854a
anyio==3.2.1
asgiref==3.4.1
async-timeout==3.0.1
attrs==21.2.0
certifi==2021.5.30
chardet==4.0.0
click==8.0.1
coverage==5.5
fastapi==0.66.0
gunicorn==20.1.0
h11==0.12.0
httpcore==0.13.6
httptools==0.2.0
httpx==0.18.2
idna==3.2
iniconfig==1.1.1
msgpack==1.0.2
multidict==5.1.0
packaging==21.0
pluggy==0.13.1
py==1.10.0
pydantic==1.8.2
PyJWT==2.1.0
pyparsing==2.4.7
pytest==6.2.4
pytest-asyncio==0.15.1
pytest-cov==2.12.1
pyzmq==22.1.0
redis==3.5.3
rfc3986==1.5.0
sanic==21.6.0
sanic-routing==0.7.0
sniffio==1.2.0
starlette==0.14.2
toml==0.10.2
typing-extensions==3.10.0.0
ujson==4.0.2
uvicorn==0.14.0
uvloop==0.15.2
websockets==9.1
yarl==1.6.3
Here is the result on MacBook 2,2 GHz 6-Core Intel Core i7, 16 GB
Ananto30 commented
Surely the benchmarks differ from machine to machine but in your case, it seems not that deviated. Can you please share your whole journey of running the benchmark, starting from setup.
skar404 commented
I was confused by the difference between Zero (Redis) and the other and failed to reproduce it
My step install and run:
# install
python3.9 -m venv venv && source venv/bin/activate
pip install -r ./benchmarks/requirements.txt
# run dependency
docker-compose up -d
# run app:
sh ./benchmarks/app_name/run_both.shdocker-compose:
version: "3.3"
services:
redis:
image: redis:latest
container_name: redis
restart: always
ports:
- "6379:6379"Ananto30 commented
Zero has a problem with signal termination. That means if you use ctrl+c, it doesn't always shut down properly. You need to kill the process using the kill command with PID or any other method that suits you. I use kill $(lsof -ti:5559) for my local tests. I Will work on this later but for now, let's focus on the benchmarks. I ran recently again and they were quite the same as readme. Could you please use a local redis, rather than the docker version? And also what did you mean by failed to reproduce it?
skar404 commented
Thanks, on local redis we managed to see that zero is faster.
Ananto30 commented
Thanks a lot for the testing. It would be great if you could share the result here. 🙏
skar404 commented
@Ananto30, my test in MacBook Pro 2018, 2,2 GHz 6-Core Intel Core i7, 16GB, local brew redis
Zero
> ./wrk -d10s -t8 -c240 http://127.0.0.1:8000/order
Running 10s test @ http://127.0.0.1:8000/order
8 threads and 240 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 58.97ms 58.78ms 479.93ms 81.53%
Req/Sec 607.07 540.51 2.27k 86.99%
47960 requests in 10.02s, 11.80MB read
Socket errors: connect 0, read 102, write 0, timeout 0
Requests/sec: 4785.87
Transfer/sec: 1.18MB
> ./wrk -d10s -t8 -c240 http://127.0.0.1:8000/order
Running 10s test @ http://127.0.0.1:8000/order
8 threads and 240 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 44.10ms 20.82ms 104.54ms 72.59%
Req/Sec 685.58 205.24 1.28k 86.12%
54612 requests in 10.01s, 13.44MB read
Socket errors: connect 0, read 100, write 0, timeout 0
Requests/sec: 5454.76
Transfer/sec: 1.34MB
> ./wrk -d10s -t8 -c240 http://127.0.0.1:8000/order
Running 10s test @ http://127.0.0.1:8000/order
8 threads and 240 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 48.67ms 36.33ms 141.08ms 72.68%
Req/Sec 682.52 275.30 1.40k 80.25%
54385 requests in 10.01s, 13.38MB read
Socket errors: connect 0, read 99, write 0, timeout 0
Requests/sec: 5432.37
Transfer/sec: 1.34MBsanic
> ./wrk -d10s -t8 -c240 http://127.0.0.1:8000/order
Running 10s test @ http://127.0.0.1:8000/order
8 threads and 240 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 72.83ms 102.84ms 1.29s 95.78%
Req/Sec 532.89 111.97 828.00 78.58%
41994 requests in 10.03s, 7.37MB read
Socket errors: connect 0, read 155, write 5, timeout 0
Requests/sec: 4186.09
Transfer/sec: 752.19KB
> ./wrk -d10s -t8 -c240 http://127.0.0.1:8000/order
Running 10s test @ http://127.0.0.1:8000/order
8 threads and 240 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 57.91ms 27.21ms 528.90ms 81.51%
Req/Sec 530.67 84.22 700.00 80.58%
42124 requests in 10.03s, 7.39MB read
Socket errors: connect 0, read 152, write 0, timeout 0
Requests/sec: 4198.46
Transfer/sec: 754.41KB
> ./wrk -d10s -t8 -c240 http://127.0.0.1:8000/order
Running 10s test @ http://127.0.0.1:8000/order
8 threads and 240 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 58.33ms 23.90ms 394.94ms 76.62%
Req/Sec 522.71 68.04 696.00 76.52%
41422 requests in 10.03s, 7.27MB read
Socket errors: connect 0, read 131, write 0, timeout 0
Requests/sec: 4128.58
Transfer/sec: 741.85KBaiohttp
> ./wrk -d10s -t8 -c240 http://127.0.0.1:8000/order
Running 10s test @ http://127.0.0.1:8000/order
8 threads and 240 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 111.69ms 175.82ms 1.68s 93.93%
Req/Sec 402.31 73.10 555.00 83.72%
31387 requests in 10.03s, 7.54MB read
Socket errors: connect 0, read 104, write 0, timeout 0
Requests/sec: 3129.75
Transfer/sec: 770.26KB
> ./wrk -d10s -t8 -c240 http://127.0.0.1:8000/order
Running 10s test @ http://127.0.0.1:8000/order
8 threads and 240 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 76.83ms 44.03ms 558.53ms 75.35%
Req/Sec 403.51 72.74 580.00 72.73%
31907 requests in 10.03s, 7.67MB read
Socket errors: connect 0, read 99, write 0, timeout 0
Requests/sec: 3180.59
Transfer/sec: 782.75KB
> ./wrk -d10s -t8 -c240 http://127.0.0.1:8000/order
Running 10s test @ http://127.0.0.1:8000/order
8 threads and 240 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 76.97ms 45.23ms 623.97ms 81.98%
Req/Sec 410.56 68.18 595.00 71.88%
32564 requests in 10.03s, 7.83MB read
Socket errors: connect 0, read 102, write 0, timeout 0
Non-2xx or 3xx responses: 1
Requests/sec: 3247.89
Transfer/sec: 799.32KBFastAPI
> ./wrk -d10s -t8 -c240 http://127.0.0.1:8000/order
Running 10s test @ http://127.0.0.1:8000/order
8 threads and 240 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 166.00ms 185.73ms 1.94s 87.89%
Req/Sec 236.01 104.55 580.00 70.42%
18729 requests in 10.07s, 3.82MB read
Socket errors: connect 0, read 102, write 0, timeout 0
Requests/sec: 1859.16
Transfer/sec: 388.54KB
> ./wrk -d10s -t8 -c240 http://127.0.0.1:8000/order
Running 10s test @ http://127.0.0.1:8000/order
8 threads and 240 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 127.85ms 75.39ms 456.28ms 66.96%
Req/Sec 239.49 70.89 404.00 71.75%
19183 requests in 10.07s, 3.91MB read
Socket errors: connect 0, read 101, write 0, timeout 0
Requests/sec: 1904.81
Transfer/sec: 398.08KB
> ./wrk -d10s -t8 -c240 http://127.0.0.1:8000/order
Running 10s test @ http://127.0.0.1:8000/order
8 threads and 240 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 134.35ms 87.76ms 750.11ms 67.57%
Req/Sec 231.22 79.52 460.00 69.62%
18536 requests in 10.09s, 3.78MB read
Socket errors: connect 0, read 101, write 0, timeout 0
Requests/sec: 1837.03
Transfer/sec: 383.91KB