[BUG] Different results on every run
Closed this issue · 6 comments
I'm running the same software against the same server mutliple times and keep getting different results. There are load balancers involved, but only on L4/TCP. TLS termination is done exclusively in a single JVM instance.
testssl.sh version 3.2.1 from https://testssl.sh/
(bcb56b7 2025-08-21 08:33:09)
Using OpenSSL 3.5.0 (Apr 8 2025) [~96 ciphers]
on Mac:/opt/homebrew/opt/openssl@3/bin/openssl
| server's cipher preferences | 9/16/25 17:43 | 9/16/25 17:44 | 9/16/25 17:45 | 9/16/25 17:46 | 9/16/25 17:47 | 9/16/25 17:50 | 9/16/25 17:59 |
|---|---|---|---|---|---|---|---|
| xc030 | yes | yes | yes | yes | yes | yes | yes |
| x9f | yes | yes | yes | yes | yes | yes | yes |
| xc09f | yes | yes | yes | yes | yes | yes | yes |
| xc02f | yes | yes | yes | yes | yes | yes | no |
| x9e | yes | yes | yes | no | yes | yes | no |
| xc09e | yes | yes | yes | no | yes | yes | no |
| x1302 | yes | yes | no | yes | yes | yes | yes |
| x1301 | no | no | no | yes | yes | yes | yes |
| x1304 | no | no | no | yes | yes | yes | no |
Hi @tewe , thanks for taking the time writing an incomplete bug report which you declared already as bug of testssl.sh w/o waiting for feedback. What do you want me to do with this information?
Test results of testssl.sh are deterministic.
Closing bc of missing information. There's a wiki which can help you further.
The [BUG] is from the New Issue form. I should have used "possible BUG". Sorry.
Further investigation showed that a testssl run with no particular arguments opens over 90 connections. Even without --parallel this happens in a somewhat short interval and triggered the per-IP rate limits on the server. Which ones were closed by the server depends on the time between connection attempts, leading to the changing results.
Since #2712 was using Cloudflare I'm pretty sure it really is a duplicate.
We'll probably switch to nmap --script ssl-enum-ciphers, because it can throttle itself.
Thanks!
Thanks for the update, @tewe and for not giving up. ;-)
I just had an allergic reaction ;-) as the bug template was not properly filled out and the [BUG / possible BUG] was reduced to the first only.
yeah, there are a lot of connections opened (and closed) during testing. We can save some but ultimately there still will be a lot. Slowing down would be possible but I am wondering how much of an edge case that would be.
I any case: Thanks for the explanation! It should be added to the FAQ. I am scratching my head whether that is related to failed github actions I tried a while back using testssl.net which is a cloudflare frontend to testssl.sh.
We'll probably switch to nmap --script ssl-enum-ciphers, because it can throttle itself
You know or have a pointer how that works?
Don't know what you were using but w/o auto throttling restructing the scan to testssl.sh -P may work better .
You know or have a pointer how that works?
nmap is a port scanner. As such it can be configured with various limits to avoid "detection". I don't think it can set them automatically.
Don't know what you were using but w/o auto throttling restricting the scan to
testssl.sh -Pmay work better .
--server-preference reduces the connections to about 30. Thanks!
I also need the particular curves though, and didn't find a switch for those.
Maybe I'll try --single-cipher and an external loop with sleep.
BTW I assume if the server doesn't support the hello it will send something back. Whereas rate limiting just closes the connection. Maybe those could be reported differently?