Openssl version not detected in current oscrypto version (pypi)
NeffIsBack opened this issue · 52 comments
Hi,
first of all thank you for your work!
We are currently encountering the Error:
oscrypto.errors.LibraryNotFoundError: Error detecting the version of libcrypto
See: https://github.com/mpgn/CrackMapExec/issues/108
I saw you already fixed that in d5f3437, but as this is not released to pypi our Tool does not fully work. Would it be possible to release a bug-fix version to pypi?
Having a bugfix release would be much appreciated we are seeing this issue on AWS Ubuntu AMIs after issuing system updates, I can imagine that there are a few less than happy AI developers out there struggling to work their way back to this issue.
We are seeing the same issue too. The OpenSSL version we are using happens to be 3.0.10. Would really appreciate it to have a new release. Thanks!
The same OpenSSL 3.0.10 1 Aug 2023 doesn't match to regex \b(\d\.\d\.\d[a-z]*)\b in _openssl/_libcrypto_cffi.py.
Last \d should be \d+ or at least \d\d? I guess.
@wbond Any chance you can look into this issue?
+1 for a bugfix release v1.3.1
+1 for a new release, this is a dependency of snowflake-connector-python, so I imagine many folks are impacted and pinning their OpenSSL version. Thanks for your time and care, we appreciate it!
The same
OpenSSL 3.0.10 1 Aug 2023doesn't match to regex\b(\d\.\d\.\d[a-z]*)\bin_openssl/_libcrypto_cffi.py. Last\dshould be\d+or at least\d\d?I guess.
I manually edited the file and changed the last \d in regex to \d+ and it worked. Note that this is a temp solution. We need to upgrade the package when a new release is available.
Any update on when this will be released?
No, this is a free-time project for me. The commit is there, so it is certainly possible to use most Python packaging tools to grab the sha of the commit.
For those too impatient, like myself... in my pyproject.toml I added in:
[tool.poetry.dependencies]
oscrypto = { git = "https://github.com/wbond/oscrypto.git", rev = "1547f53" }
And now we're all systems go. Thanks for taking the time to create this @wbond !
No, this is a free-time project for me. The commit is there, so it is certainly possible to use most Python packaging tools to grab the sha of the commit.
Thanks for the quick response, and appreciate the work you put into the package @wbond !
@connor-lough For projects that don't use pyproject.toml, is there a way you can do this via the requirements.txt file?
@connor-lough For projects that don't use pyproject.toml, is there a way you can do this via the requirements.txt file?
This is the line I added to my requirements.txt:
git+https://github.com/wbond/oscrypto.git@d5f3437
which replaced this line:
oscrypto==1.2.1
I also had to add "git" to my Dockerfile, because I was not installing git before:
apt-get install -y git
Thank you!
We are facing the same issue. Using oscrypto==1.3.0 and snowflake-connector-python==2.8.1 .
Is it possible to still face the issue despite we have hard set the versions in the requirements.txt?
@vermavikrant Yes, the problem is due to a conflict with the ocrypto library and the openssl program version 3.0.>=10 on your machine. snowflake-connector-python just happens to use the ocrypto library. The way to solve this problem is by either:
- Change the openssl version - on your machine use another version such as
1.1.1. Run$openssl versionto verify which version of openssl you are using.
Or
- upgrading your
ocryptolibrary - by pointing to the commit that includes the bug fix as mentioned above, until the patch is released.
I've spent most of the day updating our internal applications that use snowflake-connector. My fixes follow along the lines of what has been suggested, above.
If you're using Docker to build your app, you can include a pip install of the ocscrypo package prior to snowflake-connector, and it'll assume the dependency has already been met:
RUN pip install git+https://github.com/wbond/oscrypto.git@1547f535001ba568b239b8797465536759c742a3
If you need to embed it into a requirements.txt file, you can add the following line:
git+https://github.com/wbond/oscrypto.git@1547f535001ba568b239b8797465536759c742a3
If installing via setuptools.setup, then the following line should work
"oscrypto @ git+https://github.com/wbond/oscrypto.git@d5f3437"
I also had to install git into Dockerfile and Jenkinsfile, as stated above.
apt-get update && apt-get install -y git
@mwisconsin wondering why your commit hash is different? Was the fix not in d5f3437 ?
@vermavikrant I picked the latest just in case the recent commits fixed other problems I might encounter in the upcoming weeks before a release happens.
Installed it directly with
pip install -I git+https://github.com/wbond/oscrypto.git
I believe the version should be updated
@wbond really appreciate the great work you've put into this library. Could you please push a new version to PyPI so we don't need to hack it using #78 (comment)?
The reason I haven’t pushed a new release is that I’m planning a new release of asn1crypto also and this library depends on that, so I’d like to do both releases at the same time.
That release it blocked on two things:
- A test for a regression someone added, where I don’t have any real world certificate to test with and the structure is rather complex to code myself
- Deciding if the next version of asn1crypto will be a major version, which will affect if the next release of oscrypto will be a bug fix or minor version
This is the sort of bug that should have resulted in a cherry-picked bugfix release months ago when it was discovered. This kind of fix shouldn't wait on features. Everything that depends on oscrypto is actively broken once a user upgrades openssl.
Considering none of you support any of the work I do for free out of my own good will, but apparently depend on it for your job/business/product, you literally are entitled to nothing.
Complaining at me about how I spend my free time, or how I decide to prioritize my open source work will not result in me neglecting the other parts of my life for your sake.
I guess I just used up my open source energy for the day. Back to work!
I read through literally every comment here and saw zero complaining about how you spend your free time or how you prioritize your open source work. I see lots of thanks and one comment (my own) suggesting a better way to handle this kind of bug in the future. Even if you (IMO incorrectly) think this is users being ungrateful, just compare the amount of time you're likely to deal with comments and issues about this to the time a bugfix release would have taken. Just to prove I'm not ungrateful for your work, I just sent you $20. (Big spender, I know.)
@squarooticus It was you telling me that I should have cut a release instead of worry about features. In fact the dependency I spoke of is not a feature but a regression.
I appreciate your token of goodwill. I am hoping to have this sorted soon.
Thanks, and I appreciate your work!
python:3.9-slim docker image was just updated a few hours ago. and is using an incompatible openssl version as well:
OpenSSL 3.0.11 19 Sep 2023 (Library: OpenSSL 3.0.11 19 Sep 2023)
We encountered this today in Dagster Cloud and developed a workaround. We needed to install git on the system to use the git hash in setup.py.
To install git, we needed to add a dagster_cloud_pre_install.sh (don't forget chmod u+x), the contents of which are apt update -yqq && apt install -y git.
And then include the oscrypto hash mentioned upthread, in setup.py's install_requires:
"oscrypto @ git+https://github.com/wbond/oscrypto.git@1547f535001ba568b239b8797465536759c742a3",The original error was oscrypto.errors.LibraryNotFoundError: Error detecting the version of libcrypto
@asoong-94 I faced the same issue where I did not realize that Debian got rid of support for OpenSSL 3.0.9. So even if you are committing to a single SHA of the Python image, as soon as you run apt-get update in a build step, the dependency will be updated should there be an upstream dependency on the OpenSSL family of packages.
@wbond Is there an opportunity for the community to support you in getting the dependencies across the finish line or are you past the point where we might be able to offer support? I am personally just getting to the table with the libcrypto bug as it relates to the snowflake-connector-python library. I do have a workaround implemented. I can only imagine the kind of pressure you may be under given large companies like Snowflake and DataDog are dependant on this open-source library.
Last night and this morning I merged in the bugs fixes for asn1crypto. Next is to decide on the version and cut a release, followed by a release of oscrypto.
Maybe sometime tomorrow or over the weekend is my hunch.
@wbond Thank you very much for the update. We really appreciate your work 🙏🏼. I'd happily volunteer to assist you on this project.
@wbond :
Considering none of you support any of the work I do for free out of my own good will, but apparently depend on it for your job/business/product, you literally are entitled to nothing.
Complaining at me about how I spend my free time, or how I decide to prioritize my open source work will not result in me neglecting the other parts of my life for your sake.
I guess I just used up my open source energy for the day. Back to work!
Well, I hope you are not feeling criticized or pressured. I hope that everyone has been kind to you in their private messages or other channels. Open source is a lot of work and it takes a community, especially for critical low level libraries that have a lot of other projects depending on them.
I am also happy to help. Thanks for your work!
python:3.9-slim docker image was just updated a few hours ago. and is using an incompatible openssl version as well:
OpenSSL 3.0.11 19 Sep 2023 (Library: OpenSSL 3.0.11 19 Sep 2023)
Try downgrading the openssl version.
Here is an idea of what your Dockerfile might look like
FROM python:3.9
RUN mkdir /fixopenssl
WORKDIR /fixopenssl
RUN apt-get update &&\
apt-get -y remove openssl &&\
apt-get -y install build-essential zlib1g-dev &&\
apt-get -q update && apt-get -qy install wget make &&\
wget https://www.openssl.org/source/openssl-3.0.9.tar.gz &&\
tar -xzvf openssl-3.0.9.tar.gz &&\
cd openssl-3.0.9 &&\
./config --prefix=/usr/local/ssl --openssldir=/usr/local/ssl shared zlib &&\
make &&\
make install
RUN cat <<'EOT' | tee /etc/ld.so.conf.d/openssl-3.0.9.conf
/usr/local/ssl/lib64
EOT
RUN cp /usr/local/ssl/bin/openssl /usr/bin/openssl
RUN cp /usr/local/ssl/bin/c_rehash /usr/bin/c_rehash
RUN ldconfig -v &&\
update-alternatives --install /usr/bin/openssl openssl /usr/local/ssl/bin/openssl 1 &&\
update-alternatives --install /usr/bin/c_rehash c_rehash /usr/local/ssl/bin/c_rehash 1
RUN mv /fixopenssl/openssl-3.0.9/libcrypto.so.3 /usr/lib/aarch64-linux-gnu/libcrypto.so.3
CMD ["/bin/bash"]We hit the same issue with pthon:3.9-slim updated OpenSSL + snowflake-connector-python. It surprises me how big company's product relies so much on ppl's self-owned open-source project, but there's not much help coming from their side. snowflakedb/snowpark-python#992
I'll not say pushing owner for a new release is ideal, it's an open source community, let's just wait for it. For now I think just stick with the git rev solution should be enough. Thanks.
Same issue here with oscrypto 1.3.0 and OpenSSL 3.0.11
I had the same issue as well. What worked for me was adding the below to my setup.py file.
"oscrypto @ git+https://github.com/wbond/oscrypto.git@1547f535001ba568b239b8797465536759c742a3"
python:3.9-slim docker image was just updated a few hours ago. and is using an incompatible openssl version as well:
OpenSSL 3.0.11 19 Sep 2023 (Library: OpenSSL 3.0.11 19 Sep 2023)
The solution from @connor-lough didn't work for me for the above setup. It seems the fix only covers 3.0.10 (?) The only thing that worked was to hold the update of openssl in the container by adding the below to my Dockerfile:
...
RUN apt-mark hold openssl libssl3 && \
apt-get update && \
...
This keeps openssl to the following version:
root@1bda75fc2ed7:/project# openssl version
OpenSSL 3.0.9 30 May 2023 (Library: OpenSSL 3.0.9 30 May 2023)
Python debian based docker images have been updated from debian bullseye to debian bookworm during the last days. Not only the openssl version has been updated, causing this issue with the lib, the whole base system was upgraded also.
Anyone having issues using python or python-slim images, could change the image definition to use the bullseye version in the meantime: python:3.X-bullseye, python:3.X-slim-bullseye, etc.
The bullseye image versions are the ones used until the new debían bookworm was published some days ago; we can keep using them until the final fix was ready to be published, without any rush.
Let me add a workaround for a case when a system has no git (and it can't be installed):
pip install https://github.com/wbond/oscrypto/archive/d5f3437ed24257895ae1edd9e503cfb352e635a8.zipFYI. LdapRelayScan and PKINITools are also suffering from this issue. Two vital tools.
Snowflake which is a $50,000,000,000 company should be paying you something, since their tool requires your good graces. IMO.
Snowflake which is a $50,000,000,000 company should be paying you something, since their tool requires your good graces. IMO.
Unsurprisingly, Snowflake chose to move away from this lib instead: https://github.com/snowflakedb/snowflake-connector-python/blob/main/DESCRIPTION.md
Unsurprisingly, Snowflake chose to move away from this lib instead
@wbond Is there anything that a willing contributor can do to facilitate/expedite publishing a patch release to PyPI?
I work for a company that is currently affected by this issue. I well am within my rights here to dedicate some work time to helping with a PyPI release that includes this bugfix.
I can't actually benefit from Snowflake dropping your project as a dependency, because I'm stuck on an older version of their library. So you can consider me a fully captive audience.
It seems like I need to do the following:
- Commit a new entry into the the changelog describing the changes in the range
c91c86..1547f5. - Run
git tag v1.3.1to tag a new release. - Run
python run.py release.
I imagine I need both Git committer access in this repo, and contributor access to the oscrypto project on PyPI. Is that right?
Just let me know what I can do to help get this published.
Edit: It seems like there have been a lot of changes since v1.3.0 was published, and it would be a lot of work for a new contributor to sort through the commit history in order to document all of it. Would it be acceptable to cherry-pick just this one patch (+ some minimal subset of related commits, if necessary) into a separate branch to create the release? If so, what's your recommended process for that? I assume you're not interested in rebasing all of master to re-order the commits, since that would be highly disruptive to contributors and anyone using this downstream as a Git repo (e.g. vendored as a Git submodule). Maybe the authors of PRs #61, #63, #68, #76, and #77 (I think that's all of them!) would be willing to write their own changelog entries?
PSA: snowflake-connector-python 3.4.0 has been released: https://github.com/snowflakedb/snowflake-connector-python/releases/tag/v3.4.0.
any chance for a new release so that installations from pypi will include fixes such as this?
As the developer of cryptolyzer (free software), which depends on oscrypto indirectly (certvalidator), I would highly appreciate it if you could release the fix for the issue.
I'm running into this issue on an ubuntu:latest docker container running on an ARM cpu.
in my case it was fixed with:
.venv/bin/python3 -m pip install -I git+https://github.com/wbond/oscrypto.git
Is there something I can help with for there to be a new release version on pip? #81
@wbond Are there any news when the next release is planned?
I am dependent on this issue with newest Ubuntu 24.04 which comes with OpenSSL 3.0.13 and I would appreciate this fix released soon.