update cryptography pin
kapilt opened this issue · 5 comments
afaics there isn't any reason why it should be limited to versions < 40, its generally considered a core ecosystem library, and for security reasons should generally not be locked to a specific revision set. afaics the only signficant change for folks using < 40 was those building from source, given almost all users are picking up via binary wheel, this seems irrelevant to the oci package usage.
relevant line from setup.py "cryptography>=3.2.1,<40.0.0",
@kapilt we're testing the latest version of cryptography. We are not planning to remove the upper bound altogether since newer versions may contain breaking changes.
tldr - if you feel you must pin, please do so sensibly and with regular maintenance, and without sudden breakage to consumers when cryptography releases.
understood re desire to pin, but if your not going to remove the max version pin (hereafter just pin), then this requires a process for updating on fairly regular basis (multiple times a year), else we're back to the same issue reoccurring repeatedly. And users of this sdk having to file issues when cryptography releases, when all their ci and release systems break due to the hard pin when upstream releases. upstream cryptography versions 39-41 were all released this year.
surveying across other cloud providers, gcp/aws don't have direct/indirect cryptography library depends, open stack sdk doesn't pin, azure doesn't pin in its service sdks (they do a separate pkg per service, ugh), however they have a core auth lib (msal) which most other service sdks depend on which pins at <43 which accounts for an unreleased 42 , so not clear the utility, except to avoid mass breakage in sdk consumer CIs that will result as cryptography upstream releases. ie. its a real world solution for the problem of not breaking extant users/consumers when cryptography releases, and per its setup.py, its based on the actual deprecation policy for cryptography. For non provider sdks packages I generally don't see pins.
I assume by breaking changes you meant the deprecation policy, on a bug basis, its worth noting cryptography is one of the most widely tested and depended on packages as a core foundational library.
its also worth reviewing the deprecation policy for cryptography as they do X+3.
https://cryptography.io/en/latest/api-stability/#deprecation
@kapilt
We understand your frustrations with the OCI Python SDK pinning the cryptography version within a major version when you would prefer to use the latest version instead. The pinning of the dependency version was deliberate on our end after an extensive analysis of how the OCI Python SDK is used externally by our customers (with a very wide range of OS or system configurations) and by our internal teams.
We want to certify that a particular release of OCI Python SDK would work for all our customers and continues to do so over an extended period of time without requiring them to upgrade other libraries on their box. The cryptography library in its major version bumps also drops the support for OpenSSL versions (refer to changelog https://cryptography.io/en/latest/changelog/#v41-0-0). Some of our customers might be using a system where it might not be possible for them to upgrade their systems to use the newer OpenSSL versions due to restrictions in the OS they are using. In this scenario, if we leave the cryptography versions unpinned then their working setup can break. As such, we have decided to pin the dependency for cryptography to avoid such scenarios.
As for the upgrade cycle we prioritize upgrading the library proactively in cases where there have been CVE fixes. Majority of CVE fixes over the last three years were made in minor version upgrades which our customers would automatically get as we are pinning on the next major version bump only.
Also, as you have mentioned, one of our competitors does not pin cryptography while the other competitors do not have a direct pin. However, they do pin other third-party dependencies which are widely used in the Python ecosystem like requests and urllib3.
Thanks for your understanding on this topic. We understand that there are both pros and cons to the approach we've taken, but we think this path provides the best overall experience to our customers.
fwiw, I think maybe be I'm being misunderstood in some regard. doing a max pin is understandable and understood, my point is that it requires regular maintenance, and can cause potential issues again without. the upstream cryptography project has a deprecation policy that means it will be three major versions before anything gets removed, https://cryptography.io/en/latest/api-stability/#deprecation. auditing usage of cryptography library in this repo will allow better confidence in allowing for N+1|+2 versions as a max, as some other cloud sdks do to also allow max version pins on majors in the future.. Else the issue is that as soon as a cryptography major version is released, this library will become out of date, and can break CI systems due to dependency conflict.
looking at the openssl version that cryptography dropped, what was deprecated was versions older then 1.1.1d which is 4 years old has had dozens of CVEs addressed since (https://www.openssl.org/news/openssl-1.1.1-notes.html) . if this sdk library is intended to allow users of OCI to build applications that take advantage of the latest cloud resources, then holding it back because of tying to very old old and insecure system libraries is a fundamental untenable tension. which is likely to be exacerbated by recent changes in the enterprise linux ecosystem. i would suggest in that context packaging up a binary installer for older systems wrt to application usage like the cli, rather then pinning the sdk to use insecure libraries.