Add benchmarks to documentation
Closed this issue · 4 comments
aeneasr commented
JWKS endpoint (GET)
$ gobench -u "http://localhost:4444/keys/hydra.tls" -c 10000 -t 5 -auth "bearer o9RoOGORCtFeTe6HA6x0ZPKz6TikuJwDLHZSRMkPvS4.s3Q_o3uwPVfxaeAYoezvLBXgoqHFMsLnxyS0PELt-bg"
Dispatching 10000 clients
Waiting for results...
2016/07/24 18:24:11 not supported by windows
Requests: 464951 hits
Successful requests: 378513 hits
Network failed: 86438 hits
Bad requests failed (!2xx): 0 hits
Successful requests rate: 3674 hits/sec
Read throughput: 3678587 bytes/sec
Write throughput: 807390 bytes/sec
Test time: 103 sec
Warden allowed endpoint (POST)
$ gobench -u "http://localhost:4444/warden/allowed" -c 10000 -t 100 -auth "bearer PhUhkkxPqBuI0OqxrrF_UEiBs_5_HvMX8Nkn_izbFCI.hl7IbI3FJzyWtJz2XVu0a3EPgQrWjh6ff_7PofbXZtk" -d "d:\warden-allowed.txt"
Dispatching 10000 clients
Waiting for results...
Requests: 524632 hits
Successful requests: 429437 hits
Network failed: 95195 hits
Bad requests failed (!2xx): 0 hits
Successful requests rate: 4719 hits/sec
Read throughput: 1529742 bytes/sec
Write throughput: 2147564 bytes/sec
Test time: 91 sec
Warden authorized endpoint (POST)
$ gobench -u "http://localhost:4444/warden/authorized" -c 10000 -t 100 -auth "bearer PhUhkkxPqBuI0OqxrrF_UEiBs_5_HvMX8Nkn_izbFCI.hl7IbI3FJzyWtJz2XVu0a3EPgQrWjh6ff_7PofbXZtk" -d "d:\warden-authorized.txt"
Dispatching 10000 clients
Waiting for results...
2016/07/24 18:38:37 not supported by windows
Requests: 4207074 hits
Successful requests: 4200933 hits
Network failed: 6141 hits
Bad requests failed (!2xx): 0 hits
Successful requests rate: 37508 hits/sec
Read throughput: 12077688 bytes/sec
Write throughput: 14646568 bytes/sec
Test time: 112 sec
OAuth2 Client Credentials Flow
Note: This is much slower because the client credentials are validated using bcrypt.
$ gobench -u "http://localhost:4444/oauth2/token?scope=core&grant_type=client_credentials" -c 10000 -t 100 -auth "Basic YmQzZTBkZjUtYjVmYi00NzI3LWI1YzMtMzI3ODEzM2JlNzFiOlhGeWtEVUkxNlp0PD5USVQ=" -d "D:\clientcredentials.txt"
Dispatching 10000 clients
Waiting for results...
2016/07/24 18:56:54 not supported by windows
Requests: 57934 hits
Successful requests: 3448 hits
Network failed: 54486 hits
Bad requests failed (!2xx): 0 hits
Successful requests rate: 32 hits/sec
Read throughput: 10819 bytes/sec
Write throughput: 14458 bytes/sec
Test time: 105 sec
aeneasr commented
Reopening to do some proper stress testing on GCE
aeneasr commented
I did another benchmark with hydra deployed on GCE. Note that all requests used http, not https. The deployment was a single VM configured as followed (:= 80$/month):
Please note that I did not add any additional policies to the store. Also, there is an effort in ladon to greatly improve the regexp-caused CPU complexity. It is possible that future implementation will perform a lot better on warden endpoints.
In-memory
The in-memory implementation was tested.
Introspection: 500 concurrent connections
> gobench -u "http://xxx/oauth2/introspect" -c 500 -t 100 -auth "bearer y58ML1Ef0v5kT_ceVTVcaFRBeogiNzGU93tnvExqMCY.6aAJOtsH381Q6qdVoIEqy7voQvhDQ9476MMThhYpvnU" -d "D:\introspect.txt"
Dispatching 500 clients
Waiting for results...
Requests: 837084 hits
Successful requests: 837029 hits
Network failed: 55 hits
Bad requests failed (!2xx): 0 hits
Successful requests rate: 2847 hits/sec
Read throughput: 683301 bytes/sec
Write throughput: 1073268 bytes/sec
Test time: 294 sec
And CPU maxing at about 20%
Client Credentials: 100 concurrent connections
Getting client credentials is a very CPU expensive task, as we need to use bcrypt in order to receive tokens.
> gobench -u "http://xxx/oauth2/token?scope=hydra&grant_type=client_credentials" -c 100 -t 100 -auth "Basic b2NwYWRtOlFNeF9WVlF5TElRQ2E5X2t5VzRJbWExc3Aza0s5N2M4" -d "D:\clientcredentials.txt
Dispatching 100 clients
Waiting for results...
Requests: 15282 hits
Successful requests: 15282 hits
Network failed: 0 hits
Bad requests failed (!2xx): 0 hits
Successful requests rate: 63 hits/sec
Read throughput: 20972 bytes/sec
Write throughput: 20916 bytes/sec
Test time: 239 sec
CPU is drained 100%:
Warden (with token): 500 concurrent connections
gobench -u "http://xxx/warden/allowed" -c 500 -t 100 -auth "bearer y58ML1Ef0v5kT_ceVTVcaFRBeogiNzGU93tnvExqMCY.6aAJOtsH381Q6qdVoIEqy7voQvhDQ9476MMThhYpvnU" -d "d:\warden-allowed.txt"
Dispatching 500 clients
Waiting for results...
Requests: 493216 hits
Successful requests: 493177 hits
Network failed: 39 hits
Bad requests failed (!2xx): 0 hits
Successful requests rate: 2394 hits/sec
Read throughput: 299264 bytes/sec
Write throughput: 1109623 bytes/sec
Test time: 206 sec
CPU drain is similar to introspection (about 20%)
Memory snapshots
Memory usage was sampled a couple of times, nothing exceeded:
MemTotal: 9231748 kB
MemFree: 6543296 kB
MemAvailable: 8719684 kB
PostgreSQL
Please note that the PostgreSQL database was running on the same VM as hydra. PostgreSQL configuration was not modified in any way.
500 concurrent connections returned error messages, probably because postgres was used with a limit. This is why we used 100 connections instead.
The CPU drain increased due to postgres:
Introspection: 100 concurrent connections
> gobench -u "http://xxx/oauth2/introspect" -c 100 -t 10 -auth "bearer YzVVG0H3dDLnWHIxJTElUJLKneWhMhipCQE9wtgo0Co.1QBl4SokzHuUcEbfH0dJpsGY1Np45YsbrrWHVYUIE40" -d "D:\introspect.txt"
Dispatching 100 clients
Waiting for results...
Requests: 59237 hits
Successful requests: 59237 hits
Network failed: 0 hits
Bad requests failed (!2xx): 0 hits
Successful requests rate: 688 hits/sec
Read throughput: 165312 bytes/sec
Write throughput: 259426 bytes/sec
Test time: 86 sec
Client Credentials: 100 concurrent connections
> gobench -u "http://xxxx/oauth2/token?scope=hydra&grant_type=client_credentials" -c 100 -t 100 -auth "Basic b2NwYWRtOlFNeF9WVlF5TElRQ2E5X2t5VzRJbWExc3Aza0s5N2M4" -d "D:\clientcredentials.txt
Dispatching 100 clients
Waiting for results...
Requests: 1466 hits
Successful requests: 1466 hits
Network failed: 0 hits
Bad requests failed (!2xx): 0 hits
Successful requests rate: 61 hits/sec
Read throughput: 20035 bytes/sec
Write throughput: 21206 bytes/sec
Test time: 24 sec
Warden (with token): 100 concurrent connections
> gobench -u "http://xxx/warden/allowed" -c 100 -t 100 -auth "bearer YzVVG0H3dDLnWHIxJTElUJLKneWhMhipCQE9wtgo0Co.1QBl4SokzHuUcEbfH0dJpsGY1Np45YsbrrWHVYUIE40" -d "d:\warden-allowed.txt"
Dispatching 100 clients
Waiting for results...
Requests: 71513 hits
Successful requests: 71513 hits
Network failed: 0 hits
Bad requests failed (!2xx): 0 hits
Successful requests rate: 461 hits/sec
Read throughput: 57674 bytes/sec
Write throughput: 213467 bytes/sec
Test time: 155 sec
Memory snapshots
Memory usage was sampled a couple of times:
MemTotal: 9231748 kB
MemFree: 7158072 kB
MemAvailable: 8716964 kB
MemTotal: 9231748 kB
MemFree: 7244592 kB
MemAvailable: 8729828 kB
aeneasr commented
For local testing:
hey -c 500 -n 50000 -m POST -H 'authorization: Basic YWRtaW46ZGVtby1wYXNzd29yZA==' -H 'cache-control: no-cach
e' -H 'content-type: application/x-www-form-urlencoded' -d grant_type=client_credentials http://localhost:4444/oauth2
/token?grant_type=client_credentials
export T1=$(hydra token client)
export T2=$(hydra token client)
hey -c 500 -n 50000 -m POST -H "Authorization: Bearer $T1" -H 'cache-control: no-cach
e' -H 'content-type: application/x-www-form-urlencoded' -d token=$T2 http://localhost:4444/oauth2/introspect
aeneasr commented
This is resolved