[BUG] Finishing determine_optimal_proto() correctly using Alpine Linux & F5 (which maybe or not plays a role)
Closed this issue · 31 comments
I am running version 3.2.1.
One possibly related issue is #1891 which has been stale for over 4 years.
I have rebuild the docker container using both openSUSE as well as Alpine as base images, using the provided Dockerfiles. The name of the container is 'testsslpu'.
The command I use is docker run --rm -it testsslpu bcee.snet.lu.
With the openSUSE-based container, the command returns the expected output and uses the system openssl 3.1.4, not the fallback one.
While the Alpine-based container comes with openssl 3.3.3, it falls back to OpenSSL 1.0.2-bad and says
Testing with bcee.snet.lu:443 only worked using /home/testssl/bin/openssl.Linux.x86_64.
Test results may be somewhat better if the --ssl-native option is used.
I expect the Alpine container to also use the distro's openssl, which is in /usr/bin as expected.
- OS: Fedora Linux 42 (Server Edition)
- Platform: Linux 6.14.9-300.fc42.x86_64 x86_64
Thanks for filing the issue, @uselpa .
Need to look deeper into it. The point why it ended up there and the message is displayed is strange -- especially why it works running without container or with the suse container. Also if one forces /usr/bin/openssl one gets a similar message.
I expect the Alpine container to also use the distro's openssl, which is in
/usr/binas expected.
That is currently a design decision which in general should have no impact and also in your case it does not.
Educated guess is that a combination of the the server side is somewhat special so that the socket replies can't be handled by testssl and the alpine distro contributes to that . Other constellations work.
FYI: Before testssl.sh ended up there it tried several connects in which lots of elif`s were processed, and then it ended up in a situation where it cannot deal with the lack of success anymore and displays this somewhat confusing message. It has nothing to do with #1891 . That worked for me.
Todos:
- find out what the problem is with alpine for this host as opposed
to the other container and not using a container at allusing a different platform like not container or using the opensuse leap-based container - correct the misleading error message
Todos:
- find out what the problem is with alpine for this host as opposed to [...] not using a container at all
I have tried with a native install of Alpine 3.22, and the problem is the same, so it does not seem to be related to containers:
root@ssllabs ~/testssl.sh $ ./testssl.sh bcee.snet.lu
#####################################################################
testssl.sh version 3.2.1 from https://testssl.sh/
(81471c3 2025-06-15 09:48:31)
This program is free software. Distribution and modification under
GPLv2 permitted. USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!
Please file bugs @ https://testssl.sh/bugs/
#####################################################################
Using OpenSSL 1.0.2-bad (Mar 28 2025) [~179 ciphers]
on ssllabs:./bin/openssl.Linux.x86_64
Start 2025-06-23 15:54:19 -->> 185.132.60.2:443 (bcee.snet.lu) <<--
rDNS (185.132.60.2): --
Testing with bcee.snet.lu:443 only worked using ./bin/openssl.Linux.x86_64.
Test results may be somewhat better if the --ssl-native option is used.
Type "yes" to proceed and accept false negatives or positives --> ^C
root@ssllabs ~/testssl.sh $ which openssl
/usr/bin/openssl
root@ssllabs ~/testssl.sh $ /usr/bin/openssl version
OpenSSL 3.5.0 8 Apr 2025 (Library: OpenSSL 3.5.0 8 Apr 2025)
Also, explicitly specifying the openssl version does not help
root@ssllabs ~/testssl.sh $ ./testssl.sh --openssl=/usr/bin/openssl bcee.snet.lu
#####################################################################
testssl.sh version 3.2.1 from https://testssl.sh/
(81471c3 2025-06-15 09:48:31)
This program is free software. Distribution and modification under
GPLv2 permitted. USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!
Please file bugs @ https://testssl.sh/bugs/
#####################################################################
Using OpenSSL 3.5.0 (Apr 10 2025) [~96 ciphers]
on ssllabs:/usr/bin/openssl
Start 2025-06-23 16:35:28 -->> 185.132.60.2:443 (bcee.snet.lu) <<--
rDNS (185.132.60.2): --
Testing with bcee.snet.lu:443 only worked using /usr/bin/openssl.
Test results may be somewhat better if the --ssl-native option is used.
Type "yes" to proceed and accept false negatives or positives --> ^C
ok, good point, thx
I've been wondering whether this might be linked to openssl's build options.
For Alpine 3.22, they can be found here, but I was unable to find the equivalent for openSUSE.
It also depends on the site, because for example this works (but leaving out --openssl= falls back to openssl-bad)
root@ssllabs ~/testssl.sh $ ./testssl.sh --openssl=/usr/bin/openssl apple.com
#####################################################################
testssl.sh version 3.2.1 from https://testssl.sh/
(81471c3 2025-06-15 09:48:31)
This program is free software. Distribution and modification under
GPLv2 permitted. USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!
Please file bugs @ https://testssl.sh/bugs/
#####################################################################
Using OpenSSL 3.5.0 (Apr 10 2025) [~96 ciphers]
on ssllabs:/usr/bin/openssl
Start 2025-06-23 17:46:32 -->> 17.253.144.10:443 (apple.com) <<--
Further IP addresses: 2620:149:af0::10
rDNS (17.253.144.10): world-any.aaplimg.com. www.brkgls.com. brkgls.com. apple.com.au. podcast.apple.com. seminars.apple.com. apple.com.pa. asia.apple.com. apple.com.gy. apple.fr. icloud.com.
aperturetrialbuy.apple.com. apple.com.do. apple.co.uk. apple.com.my. apple.com.co. firewire.apple.com. apple.com.cn. apple.com.sg. applecomputer.co.kr.
applejava.apple.com. apple.com.pe. appstore.com. livepage.apple.com. itunespartner.apple.com. apple.com.bo. apple.com.tt. applescript.apple.com. apple.com.uy.
apple.com.hn. vipd-healthcheck.a01.3banana.com. apple.com.lk. apple.com.ai. apple.it. iphone.apple.com. advertising.apple.com. apple.es. apple.nl. apple.de.
iworktrialbuy.apple.com. apple.com. apple.com.py. guide.apple.com. apple.com.mx. squeakytoytrainingcamp.com. shake.apple.com. apple.ca.
Service detected: HTTP
Testing protocols via sockets except NPN+ALPN
[...]
I lack expertise for Alpine but there were a lot of other problems in the past , that's why we use for docker hub something different, kudos to @polarathene . The problems were mostly related to musl libc.
But what might be worth a shot is to install the gcompat package. It doesn't help the users though.
In the end though determine_optimal_proto() needs to be debugged and figured out why any connection with bash sockets failed , see elif statement around L22740.
I tried gcompat and unfortunately that didn't help.
If you can give me a clue on how to help you please don't hesitate!
root@ssllabs ~/testssl.sh $ apk info gcompat
gcompat-1.1.0-r4 description:
GNU C Library compatibility layer for musl
gcompat-1.1.0-r4 webpage:
https://git.adelielinux.org/adelie/gcompat
gcompat-1.1.0-r4 installed size:
87 KiB
root@ssllabs ~/testssl.sh $ ./testssl.sh --openssl=/usr/bin/openssl snet.lu
#####################################################################
testssl.sh version 3.2.1 from https://testssl.sh/
(81471c3 2025-06-15 09:48:31)
This program is free software. Distribution and modification under
GPLv2 permitted. USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!
Please file bugs @ https://testssl.sh/bugs/
#####################################################################
Using OpenSSL 3.5.0 (Apr 10 2025) [~96 ciphers]
on ssllabs:/usr/bin/openssl
Start 2025-06-23 18:28:21 -->> 185.132.60.2:443 (snet.lu) <<--
rDNS (185.132.60.2): --
Testing with snet.lu:443 only worked using /usr/bin/openssl.
Test results may be somewhat better if the --ssl-native option is used.
Type "yes" to proceed and accept false negatives or positives --> ^C
root@ssllabs ~/testssl.sh $ ./testssl.sh snet.lu
#####################################################################
testssl.sh version 3.2.1 from https://testssl.sh/
(81471c3 2025-06-15 09:48:31)
This program is free software. Distribution and modification under
GPLv2 permitted. USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!
Please file bugs @ https://testssl.sh/bugs/
#####################################################################
Using OpenSSL 1.0.2-bad (Mar 28 2025) [~179 ciphers]
on ssllabs:./bin/openssl.Linux.x86_64
Start 2025-06-23 18:28:51 -->> 185.132.60.2:443 (snet.lu) <<--
rDNS (185.132.60.2): --
Testing with snet.lu:443 only worked using ./bin/openssl.Linux.x86_64.
Test results may be somewhat better if the --ssl-native option is used.
Type "yes" to proceed and accept false negatives or positives --> ^C
Thanks!
Is there a glibc as a package under Alpine?
As a datapoint, I tried the same on a Debian testing installation (no containers involved). Works just fine.
I was trying to create an Alpine-based container with my code and testssl.sh which it uses, as I do not feel at ease with openSUSE. But I use both Alpine and Debian, so the latter will do in case this is not fixable.
Thanks. Normally if I get bug reports like yours I try Debian, OpenSuse, MacOS first and some Red-Hat-ish distro. As it is the "natural" baseline when trying to find out what's wrong I didn't mention that.
Using gcompat requires LD_PRELOAD I think?
However when I last tried gcompat with another project (deno), it didn't cover all glibc symbols required to function. You would call openssl with LD_PRELOAD prepended, so you might need to make a wrapper script that testssl calls (don't want to prepend to testssl command since that'd affect all native musl commands I think), forwarding any options / input.
Alternatively if you can build the old openssl for musl too?
Another option is to bring over all dependencies openssl has, as defined by ldd, then you could use patchelf to set an rpath or runpath (latter requires the same modification to all direct deps to resolve any trabsitive deps, while Rpath is recursive). But IIRC this old openssl from testssl is a static build, so you'd need the equivalent dynamic link build instead.
Doing it this way will be a portable glibc without the static glibc caveats I think? (well dlopen() calls might be problematic if they're for additional libraries that are provided at the rpath). Size might be larger too.
Personally I discourage using Alpine as the motives for choosing it often aren't worth the troubleshooting tradeoff whentthings go wrong 😅 After all I got the difference down to 27MB (Alpine) vs 54MB (openSUSE or Ubuntu-chiseled).
It's possible to reduce glibc further but testssl.sh would need to support some alternative programs that are statically linked IIRC (main issue is large package weight to support some commands).
Hello @polarathene
I don't think that the solution here is to turn Alpine into an entirely different distro, especially for Alpine users who want to use testssl.sh outside of a container. In that case I would rather remove the Alpine Dockerfile from the repo and adapt some text which as of now says that testssl.sh runs on any Linux distro.
I'd much rather understand where the incompatibility stems from. Since coreutils are installed and appear to work properly, and testssl.sh does not use glibc directly (I assume), understanding at what point there is a behavioural difference on Alpine is likely the most promising path to understanding.
Regards,
-Patrick
I'd much rather understand where the incompatibility stems from
We're at least two ;-)
testssl.sh does not use glibc directly (I assume),
It's a library @uselpa , so that's always true. :-) Any libc is used for core operations like file or network. And for Alpine, as @polarathene pointed out, it caused some pain for troubleshooting to nail them down.
Thus the midterm term goal is to replace Alpine for this project. But we maybe should find out what's wrong here first
@uselpa : As you offered your help....
I would need to install a Alpine VM first and compare each steps in determine_optimal_proto() with e.g. a standard distro like Debian. By each steps I mean putting set -x in the beginning and set +x at the end of determine_optimal_proto() . First on Alpine and to maybe get a side-by-side comparison the same with e.g. any Debian version.
If you feel like you could assist, that would be awesome. If not, that's fine too.
Something like this?
./testssl.sh --openssl=/usr/bin/openssl --color=0 snet.lu > determine_optimal_proto.txt 2>&1
determine_optimal_proto.txt
@uselpa I am not saying to change Alpine into different distro by changes. The openssl that testssl adds is a custom version compiled to test extra ciphers AFAIK, and it statically links glibc, which has caveats as this does not do proper static linking if calls like dlopen() are made (static binaries should not do this, it will cause problems).
My comment discussed ways to make it possible to run in alpine with dynamic linking, bundling it's dependencies and ensuring it uses those alone, not anything on alpine.
testssl.sh itself would continue to run natively with musl and everything else in Alpine, it would only be the custom openssl that would differ. The other option is to build the custom openssl for musl too, rather than copying the glibc version over and hoping it works without issues 😅
Thanks for the explanation @polarathene, that makes more sense to me now.
I'm conflicted about the custom openssl version provided by testssl.sh since I have no visibility into how up-to-date it is, while on the other hand distros put efforts into providing a compatible version with their distro. So maybe, instead of bundling a local version, one could
- at the startof testssl.sh, warn about potential insufficiencies of the OS version of openssl
- if necessary and applicable, provide an alternative version in a distro-specific repository, or even ask upstream to do it, but based on the distro-provided version.
Which is why I wanted to compare the build options of openssl, see #2807 (comment)
@polarathene : I´d rather spend my time differently than maintaining a yet another openssl version. Besides it looks also that using /usr/bin/openssl makes problems, see above . Not sure yet whether the dynamically linked openssl is to blame or something else. This is TBD.
Looks more like a logic error.
Something like this?
./testssl.sh --openssl=/usr/bin/openssl --color=0 snet.lu > determine_optimal_proto.txt 2>&1
determine_optimal_proto.txt
Sorry, my bad, the real culprit seems determine_optimal_sockets_params() . Could you set -x the start of the function (end before return set +x) and pass the output to me, please? Best would be after a fresh git pull.
Thanks @uselpa !
This is more difficult than assumed: for bash sockets the reply from your output is empty, so as a consequence ALL_FAILED is true and the last elif in determine_optimal_proto() catches that correctly with the error message (which could be improved).
FYaI: As the message says: testssl.sh needs bash sockets to work. If it doesn't, it quits. Strangely it does that only on Alpine (and likely for a few hosts, otherwise I would have gotten more complaints)
This is a vim diff view. LHS is the debug out put for not Alpine, RHS is Alpine. The green cross shows the client hello which is exactly the same. The blue cross shows the server reply is empty.
@uselpa : You have relationship with that site? Or can you tell using Alpine / the docker alpine image for which type of hosts that happens?
I don't think we should concentrate on the websites since just about any distro seems to be able to manage it except Alpine. And the combination of Alpine with bash and coreutils and a socket-oriented bash program may not be the most common combination and therefore a well-tested path. But at this point I don't have enough information to open a bug against Alpine.
Also, the sockets work on Alpine for most sites, just not for some. This looks like we hit an edge case, either in your code or on some Alpine component (openssl, kernel, the bash or coreutils port).
Do you think you could create a minimal bash example that fails on Alpine when it shouldn't? I can set up a test environment I can give you access to if this helps.
All this does not seem linked to the fact that /usr/bin/openssl is apparently not found, when it's at the same location as in other distros. Easily circumvented, but somehow intriguing too.
Personally, I set up a container based on Fedora with testssl and my code in a container and it just works fine (on Alpine). So if you think it's not worth it, I'm fine - I'd just suggest then to put a warning that Alpine compatibility is not 100%.
OTOH, if you want to dig into it, I'm game.
Basically you're right as I have more on my to do list for this project than I can probably handle.
OTOH I would rather get a feel how much of this is an edge case and also an understanding what the culprit is. Maybe you can help.
I agree the combination of F5 on the server and Alpine on the client side maybe aren't often used. But is that all? Or is it less, maybe because of a configuration issue of the sites here. Or is it a firmware issue, that only certain versions of F5 loadbalancers are affected?
What you can do if you want to assist is trying to play with the devel options:
DEBUG=5 ./testssl.sh --devel 03 "c0,28, c0,27, c0,2f, c0,30" snet.lu
and compare e.g. Fedora and Alpine. The 4 hexcodes with 16 bits are ciphers which the site supports. "03" is the only protocol supported (TLS 1.2).
If one looks at the screenshot above it's strange, because the only potential culprit on the client side is dd or hexdump. But still I am wondering: where does the role of the server side makes the difference? Both distros send exactly the same bytes over the wire, see socksend_clienthello(). At least in the screenshot we only see that parsing is different.
There are other sites which I guess are behind the F5 which seem to work, such as my.spuerkeess.lu. Of course the latter might be configured differently. But again, they all work on Fedora.
debug5-alpine.txt
debug5-fedora.txt
debug5-alpine-my.txt
Regarding dd and hexdump:
On Alpine, hexdump is still the busybox version
root@ssllabs ~/testssl.sh $ dd --version
dd (coreutils) 9.7
root@ssllabs ~/testssl.sh $ hexdump
BusyBox v1.37.0 (2025-05-26 20:04:45 UTC) multi-call binary.
Usage: hexdump [-bcdoxCv] [-e FMT] [-f FMT_FILE] [-n LEN] [-s OFS] [FILE]...
Display FILEs (or stdin) in a user specified format
-b 1-byte octal display
-c 1-byte character display
-d 2-byte decimal display
-o 2-byte octal display
-x 2-byte hex display
-C hex+ASCII 16 bytes per line
-v Show all (no dup folding)
-e FORMAT_STR Example: '16/1 "%02x|""\n"'
-f FORMAT_FILE
-n LENGTH Show only first LENGTH bytes
-s OFFSET Skip OFFSET bytes
whereas on Fedora
root@fed:~/testssl.sh# dd --version
dd (coreutils) 9.6
root@fed:~/testssl.sh# hexdump --version
hexdump from util-linux 2.40.4
so since there is a hexdump package in Alpine (https://pkgs.alpinelinux.org/contents?file=hexdump&path=&name=&branch=v3.22&repo=&arch=) I installed that one
root@ssllabs ~/testssl.sh $ apk add hexdump
fetch https://dl-cdn.alpinelinux.org/alpine/v3.22/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.22/community/x86_64/APKINDEX.tar.gz
(1/1) Installing hexdump (2.41-r9)
Executing busybox-1.37.0-r18.trigger
OK: 542 MiB in 319 packages
root@ssllabs ~/testssl.sh $ hexdump --version
hexdump from util-linux 2.41
but that didn't help, unfortunately (although the debug output is slightly different)
Hi @uselpa ,
thanks for helping. I looked at the outputs you provided but other than empty server reply or TLS alert (debug5-alpine.txt) the server side doesn't want to talk to Alpine clients.
I searched for some obvious BIGIPs via $hacker_search_engine, tested my self with the Alpine image those. All ~10 worked.
Bottom line: I am running out of ideas . Actually we don't even know whether it's F5/BigIP which plays a role here when used with an Alpine test client. That is just an assumption, as most of those pieces do TLS offloading or maybe WAF functions (ASM).
Thus, I am considering this now as an edge case and unless one comes up with new ideas I'd rather focus on different tasks here.
Hi @drwetter,
that's OK for me.
As a final datapoint, I installed a GUI on an Alpine test machine and tried to connect to the site in question with Firefox (it's a web-banking solution for a bank I have an account with).
I did the test both with a pristine install, and one with coreutils and the other packages required for testssl.sh installed.
Both times I managed to authenticate and to connect interactively without problems nor any kind of warnings .
Thanks for your feedback, and many thanks for maintaining the project!
-Patrick