cryptography with OpenSSL 3.x
Closed this issue · 18 comments
I'm using the cryptography module and trying to use the latest OpenSSL 3.3.0, then I encountered this error when trying to use cryptography:
RuntimeError: OpenSSL 3.0's legacy provider failed to load. This is a fatal error by default, but cryptography supports running without legacy algorithms by setting the environment variable CRYPTOGRAPHY_OPENSSL_NO_LEGACY. If you did not expect this error, you have likely made a mistake with your OpenSSL configuration.
After some debugging, I found out that the legacy modules are built in a separate shared object called legacy.so, in my case it is located in /usr/lib/ossl-modules/legacy.so, and PyInstaller is not collecting it.
Can someone point me out what will be the best way to implement it? should I change the hook-cryptography.py or create a new one?
Do you actually need those legacy algorithms? If you don't, it might be better to just set that CRYPTOGRAPHY_OPENSSL_NO_LEGACY environment variable before importing cryptography...
Otherwise, cryptography hook would need to explicitly look for these additional OpenSSL shared libraries (and account for differences between distributions).
@rokm Unfortunately, I still need them 😥
Do you have any suggestions on how to locate them without hardcoding the full path?
I tried using collect_dynamic_libs but I can't figure it out as it is not able to locate it
collect_dynamic_libs is for shared libs that are bundled within python packages, so it won't work with this.
Either you will need to look into bunch of standard hard-coded locations, or try to determine the location of libssl.so.3 (either using ctypes helpers, or using PyInstaller.depend.bindepend.resolve_library_path), and then look for ossl-modules directory next to it.
On my Fedora system, this would be /lib64/libssl.so.3 and /lib64/ossl-modules/legacy.so; on debian-based systems, it seems to be /usr/lib/x86_64-linux-gnu/libssl.so.3 and /usr/lib/x86_64-linux-gnu/ossl-modules/legacy.so, etc.
I need to check how this is handled on Windows and macOS, though. And actually, how does OpenSSL determine the search path (is it also using relative search path, or does it have hardcoded absolute search location, that might be overridden via environment variable)...
The fastest way for your application build right now is to hard-code the path for your build system (and you can do that in the .spec file instead of the hook), though, until more generic solution is added to the hook.
Thanks, I think that this is what I will do for now, would appreciate any help getting it into the official hooks 😃
@rokm I added the file in my spec file and I do see it collected inside the application directory but this still not working, looking at strace output I see that it tries to load it from the system path and not from the application directory
openat(AT_FDCWD, "/usr/lib64/ossl-modules/legacy.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
Is there something I need to do to fix the path?
I think the search location can be manipulated via OPENSSL_MODULES environment variable; can you first try to set it manually outside of the executable? And if that works, then dynamically/programmatically at the very top of your program?
Thanks, that seems to do the trick, I will go with adding a "runtime hook" that will set this environment variable for now until we will be able to work around this issue
@amirrossert Can you check if #727 works for you?
@rokm Thanks, I will try it today and update with the result
@rokm I just tested it and it seems to be working as expected, I only tested it on Linux for now
@rokm Thanks I have tried this new hook on my Alpine build and it seems that it fails to find the ossl-modules module directory, I have debugged this a bit and it seems that the issue is with how bindepend.resolve_library_path is implemented, in Linux it finds the lib directory using ldconfig which is not really implemented in Alpine.
I noticed that the _which_library fails since it expects to get the name without the .so suffix (currently it is getting libssl.so.3)
name:
The library name including the `lib` prefix but excluding any `.so` suffix.
What I tested is this change in the hook and it worked, do you think this is the way to go?
lib_name = 'libssl.so.3' if not compat.is_musl else 'libssl'
I guess that this should also work for non Alpine Linux systems so maybe the if is not needed but we can always use libssl (or try both if one fails)
(I also noticed that the compat module does not have is_alpine property which can be handy in this cases 😃)
@amirrossert Thanks for the feedback!
This means that combination of _resolve_library_path_unix and _which_library is broken in terms of expected input name, and this is masked by the fact that most of the time, we end up using the ldconfig branch in _resolve_library_path_unix.
What I tested is this change in the hook and it worked, do you think this is the way to go?
lib_name = 'libssl.so.3' if not compat.is_musl else 'libssl'
As a work-around, yes, although not in exactly that form - because it will break again once _resolve_library_path_unix is fixed. Instead, we could do something like this:
# Linux and other POSIX systems
lib_name = 'libssl.so.3'
openssl_lib = bindepend.resolve_library_path(lib_name)
if openssl_lib is None and compat.is_musl:
# Work-around for bug in `bindepend.resolve_library_path` in PyInstaller 6.x, <= 6.6
lib_name = 'libssl'
openssl_lib = bindepend.resolve_library_path(lib_name)@rokm Thanks, do you want me to create a PR to fix that or you are already on it?
Also, will you fix the resolve_library_path flow?
@rokm Thanks, do you want me to create a PR to fix that or you are already on it?
You can open a PR if you want. Otherwise, I'll do it later today or tomorrow.
Also, will you fix the
resolve_library_pathflow?
Yeah, I'll look into fixing the resolve_library_path.
Hmm, actually, while we now correctly resolve the location of libssl.so.3, at least in my alpine container, the true location is /lib/libssl.so.3, while the ossl-directory is in /usr/lib, along with symlink for libssl.so.3.
Is it the same on your test system, or do you have a different configuration there (either merged /lib and /usr/lib, or true location of libssl.so.3 being in /usr/lib)?
@rokm Actually I have a custom docker image as a build system where I compile OpenSSL from the source code and install which ends up in /usr/lib/libssl.so.3
If I take a fresh Alpine image and install openssl I do see that it is located both in /lib and /usr/lib
/ # find / -name libssl.so.3
/lib/libssl.so.3
/usr/lib/libssl.so.3