setup.py: SSL certification fails
GFleishman opened this issue · 17 comments
Hi guys,
Attempting to install pyklb via instructions:
cd ~/source/pyklb
pip install wheels
python setup.py bdist_wheel
pip install /path/to/wheel.whl
Step 3 above (setup.py) fails with:
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)
and a second unhandled exception:
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)
Presumably, setup.py is reaching out to some server to download the most appropriate precompiled binaries for the KLB library for my system (MacOS 10.13.3), but this fails SSL certification. The KLB repository actually doesn't seem to include binaries for MacOS, and I'd rather try to fix this SSL issue than compile KLB myself.
Are you aware of any reason on your end (i.e. not having to do with my environment/network) that would cause SSL cert to fail?
I can not reproduce the problem on Linux or Windows and I don't currently have good access to MacOS, although I'm not sure whether a SSL certificate issue would not in fact be OS-independent.
You are correct, the install script tries to download a precompiled binary of libklb for your OS. Sadly, this turned out to be easier than making Python packaging, cython and CMake cooperate smoothly. The file is downloaded from a bitbucket repository (the main KLB repo listed in the readme). Could it be that you have a bitbucket account with configured SSL keys? Do you think it could be erroneously using your keys when trying to access the KLB repo?
If I knew where in the repo (https://bitbucket.org/fernandoamat/keller-lab-block-filetype/src/8e3c39b750225bf089ee5f08ac190e763e22e4fc?at=master) the appropriate binary was (and downloaded it), and where it is expected to be on my system (an put it there), would setup.py just forgo trying to download and proceed? I don't really need to solve the SSL issue right now, just want to get pyklb installed :).
That's totally reasonable and should work. The binary is saved to /path/to/pyklb/build/lib/libklb.dylib. You'll also have to deactivate the download in setup.py. Commeting out the entire if-else-clause starting with if platform.architecture()[0].startswith("64"): should work. Please let me know if not or if you need any help.
After compiling the cython code as part of executing setup.py, the pyklb module can be un-/installed with pip, but please note that it still depends on libklb, which is not handled by pip. You have to copy it to the site-packages subfolder of your python install manually (and delete it manually to uninstall).
Best of luck, let me know if it works, and especially if not.
I just discovered that it actually is a bit more complicated.
First of all, I forgot there is an option to not download the library. Instead of modifying setup.py, You can try python setup.py bdist_wheel --skip-klb-download. For this to work, you should manually download libklb.dylib to /path/to/pyklb/build/lib/ and common.h and klb_Cwrapper.h to /path/to/pyklb/build/include/
I also just now found and fixed a bug regarding this in setup.py. Sorry for that.
Pulled and this compile error is now gone (thanks!), but it seems like there is a mismatch between the expected library and the one I've got. The only one in the KLB repo was:
build/lib:
libklb-jni.dylib
(That was the only *.dylib file in the KLB repo, it was in the javaWrapper file tree)
Now if I run setup I get:
gcc -bundle -undefined dynamic_lookup build/temp.macosx-10.13-x86_64-3.6/src/pyklb.o -Lbuild/lib -lklb -o build/lib.macosx-10.13-x86_64-3.6/pyklb.cpython-36m-darwin.so
ld: library not found for -lklb
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'gcc' failed with exit status 1
You may have to rename libklb-jni-dylib to libklb.dylib. Not really clean but should work.
That gets the setup script to finish and produce a .whl file, which pip supposedly installs successfully, but when I go to python and try to import I get:
ImportError: dlopen(/Users/fleishmang/Development/bin/py364/lib/python3.6/site-packages/pyklb.cpython-36m-darwin.so, 2): Library not loaded: /Users/clackn/src/keller-lab-block-filetype/build/libklb-jni.dylib
Referenced from: /Users/fleishmang/Development/bin/py364/lib/python3.6/site-packages/pyklb.cpython-36m-darwin.so
Reason: image not found
So something is still funny with the location/naming of the library.
Seems like maybe I should try to resolve the SSL issue (but it also maybe seems like something has changed w.r.t. which binaries are available in the KLB repo?), or try to compile the KLB library from source and proceed with the --skip-klb-download option again?
The exact file you need is this one:
https://bitbucket.org/fernandoamat/keller-lab-block-filetype/raw/5edcaecc858911c7b3855579bde5cb3116cb4680/bin/libklb.dylib
Can you download it from your webbrowser?
Yes, something has indeed changed with respect to the available binaries in the main KLB repo. I didn't notice that we removed the bin subfolder, which is where they used to live. The download link above is the exact one used by setup.py and it goes back to an older correct commit (where this folder still existed). So it should still work, if you can somehow get the file by either downloading it through the link or cloning the repo and checking out commit 5edcae. Apologies for that. I'll look again into building libklb from source when invoking setup.py. That is ultimately the only good solution to this problem.
Hi again (sorry for lag, had meeting; also, thanks for all the help with this so far!).
I was able to download the library via the link - didn't think of checking an older commit for the bin folder, that was a good idea! - still able to run setup.py successfully, but not able to import the library:
In [1]: import pyklb
ImportError Traceback (most recent call last)
in ()
----> 1 import pyklb
ImportError: dlopen(/Users/fleishmang/Development/bin/py364/lib/python3.6/site-packages/pyklb.cpython-36m-darwin.so, 2): Library not loaded: /Users/clackn/src/keller-lab-block-filetype/build/libklb.dylib
Referenced from: /Users/fleishmang/Development/bin/py364/lib/python3.6/site-packages/pyklb.cpython-36m-darwin.so
Reason: image not found
Seems like python is looking for the library here:
/Users/clackn/src/keller-lab-block-filetype/build/libklb.dylib
which seems like a directory belonging to Nathan Clack? Not sure why python would be thinking to look there unless that path is hardcoded somewhere in either pyklb or klb?
Did you copy libklb.dylib to /Users/fleishmang/Development/bin/py364/lib/python3.6/site-packages/?
Originally I just had it in /path/to/pyklb/build/lib
Just tried copying it to:
/Users/fleishmang/Development/bin/py364/lib/python3.6/site-packages
But I get the same error when importing.
I also tried:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/Users/fleishmang/Development/bin/py364/lib/python3.6/site-packages
then reload python and import, but same error.
I may try compiling KLB from source, then following through with the rest of your installation procedure.
No idea. I'm not a MacOS user. We are not hard-coding any file paths. The dylib needs to be at /path/to/pyklb/build/lib/ during compilation and at /path/to/python/site-packages/ when actually importing in Python. I do not know why MacOS is looking for it at the path that @nclack compiled it.
@nclack - any ideas?
Hi, just saw this.
I'm still a bit confused about this, and I don't have a mac to test on, but:
I believe this is an osx specific issue related to use of the install path, or something similar. One fix might to alter the klb build to set the @rpath appropriately (probably something like $ORIGIN). If that sounds like a lot of magic words to you, that's because they are magic. But they aren't too bad, and there may be another way.
Dynamic loading is only involved because the pyklb cython extension is linking against the klb dylib, a dynamic library. If pyklb could link against the static klb library this wouldn't be a problem. In turn, one would also need to link against the dependencies of klb: libz and libbz2.
I hope that helps,
Nathan