Dependency on rust removes support for a number of platforms
mgorny opened this issue · 179 comments
I would like to report that the newly added dependency on Rust has made it impossible to package cryptography for a number of supported Gentoo architectures (and these are architectures where people actually use Python packages that depend on cryptography).
Please see Platform Support. Besides degrading a few of our targets to 'Tier 3', Rust completely does not support at least alpha, hppa, ia64, m68k, s390 (not -x). The cost of adjusting cryptography's C code to these platforms (even if we assumed they wouldn't work out of the box as they do so far) is much less than the cost of porting the whole Rust. You can't really expect volunteers to port the whole Rust to other architectures just to get cryptography back.
Right now it's a really hard struggle to even get Rust packaged for non-Tier 1 architectures (i.e. where upstream does not provide bootstrap binaries).
destroyed my CI/CD Pipeline this morning 😆
It is also impossible to build on ANY alpine linux version/architecture. Please remove this dependency as soon as possible.
It is also impossible to build on ANY alpine linux version/architecture.
Update to most recent Alpine with Rust >= 1.45. Alpine latest has rust (1.47.0-r2).
Please remove this dependency as soon as possible.
Not going to happen. Rust dependency will stay to replace C code with an actual safe language.
You can disable Rust integration in 3.4.x (see FAQ). Starting with 3.5 cryptography will have mandatory Rust code.
So the only "reason" for this change is to use a "actual safe language"? ...ahhh...yeah...
But blowing up systems with unnecessary software packages is a more secure way, that's for sure...
(my 2 ¢)
I guess this means we'll have to fork Cryptography, and render the original package irrelevant.
So the only "reason" for this change is to use a "actual safe language"? ...ahhh...yeah...
Yes, memory safety is a very compelling reason to use Rust. C is a bad language to implement parsers for e.g. ASN.1. Check out @alex and @reaperhulk side-project https://twitter.com/LazyFishBarrel
But blowing up systems with unnecessary software packages is a more secure way, that's for sure...
(my 2 ¢)
The new Rust code adds exactly 0 (zero) runtime packages to Cryptography. Rust, Cargo, pyo3, its dependencies, and setuptools_rust are build-time dependencies only.
Pinned to second last version to get our pipelines up and running again, at least until I've figured out what should be done long term
Memory safety is of course a compelling reason, but you should also keep in mind that may developers out there are using your package, which is great by the way, and are now forced to upgrade all their containers, ci/cd pipelines, packages and so on. A little piece of information upfront or a deprecation warning would have helped a lot. Or maybe I just overlooked something.
Pinned to second last version to get our pipelines up and running again, at least until I've figured out what should be done long term
Yes, that's exactly what I did.
I think that it is clear that you have broken semantic versioning with both the python 2 change, and the requirement for rust at build time. From the changelog:
BACKWARDS INCOMPATIBLE: Support for Python 2 has been removed.
We now ship manylinux2014 wheels and no longer ship manylinux1 wheels. Users should upgrade to the latest pip to ensure this doesn’t cause issues downloading wheels on their platform.
cryptography now incorporates Rust code. Users building cryptography themselves will need to have the Rust toolchain installed. Users who use an officially produced wheel will not need to make any changes. The minimum supported Rust version is 1.45.0.
cryptography now has PEP 484 type hints on nearly all of of its public APIs. Users can begin using them to type check their code with mypy.
This should probably be a 4.0 release it would cause a lot less pain for the community. I suppose many have cryptography>=3.0,<4 in setup.py or requirements.txt.
Update
I was unaware that you don't use semantic versioning. I have found in the docs https://cryptography.io/en/latest/api-stability.html#versioning. However it was a little confusing given the style. As many of our dependencies require this we will try not to pin a version and install rust/use manywheel packages.
A little piece of information upfront or a deprecation warning would have helped a lot. Or maybe I just overlooked something.
Looks like you have missed all announcements and FAQ entries.
Rust bindings kicked off in July 2020 with #5357. Alex started a thread on the cryptography developer mailing list in December to get feedback from packagers and users. The FAQ also contains instrutions how you can disable Rust bindings.
PS: I've been preparing Fedora packaging for almost three months. My Fedora Rawhide packages of python-cryptography 3.4 landed about 12 hours after the release -- and I had to jump through several extra hoops because Fedora does not permit vendoring of Rust crates.
Yes, memory safety is a very compelling reason to use Rust. C is a bad language to implement parsers for e.g. ASN.1. Check out @alex and @reaperhulk side-project https://twitter.com/LazyFishBarrel
I'm sure all the users who aren't able to use cryptography anymore will appreciate the memory safety.
Rust bindings kicked off in July 2020 with #5357. Alex started a thread on the cryptography developer mailing list in December to get feedback from packagers and users. The FAQ also contains instrutions how you can disable Rust bindings.
Do you seriously expect people who maintain a few hundred Python packages to follow development mailing lists of every single one of them?
I'm sure all the users who aren't able to use cryptography anymore will appreciate the memory safety.
The majority of users either uses binary wheels (macOS x86_64, glibc Linux x86_64 + aarch64, Windows X86 + X86_64) or Linux distro packages. Binary wheels don't require an additional Rust libraries. Only users on Alpine (musl), BSD, other hardware platforms, and distro packagers are affected.
Do you seriously expect people who maintain a few hundred Python packages to follow development mailing lists of every single one of them?
Do you have constructive suggestions how to communicate changes additionally to project mailing lists, Github issue tracker, project IRC channel, documentation (changelog, FAQ), and Twitter channels?
I wonder if there's a way to produce a more helpful error message, perhaps with a link to some documentation about the new required build deps. The current message most people will hit is
=============================DEBUG ASSISTANCE=============================
If you are seeing a compilation error please try the following steps to
successfully install cryptography:
1) Upgrade to the latest pip and try again. This will fix errors for most
users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
2) Read https://cryptography.io/en/latest/installation.html for specific
instructions for your platform.
3) Check our frequently asked questions for more information:
https://cryptography.io/en/latest/faq.html
4) Ensure you have a recent Rust toolchain installed.
=============================DEBUG ASSISTANCE=============================
Which does sort of attempt to guide people the right way but could be more specific for common cases like Alpine.
Trying to follow the docs as an Alpine user who doesn't know about Rust is kind of confusing right now. As the Alpine specific steps only tell you to install gcc but as you point out above latest Alpine actually has a new enough rust in APK.
The majority of users either uses binary wheels (macOS x86_64, glibc Linux x86_64 + aarch64, Windows X86 + X86_64) or Linux distro packages. Binary wheels don't require an additional Rust libraries. Only users on Alpine (musl), BSD, other hardware platforms, and distro packagers are affected.
That's a very roundabout way of showing the middle finger to users of 'other hardware platforms' that are not supported by Rust (or LLVM) at all.
CI testing went down the drain with this in GitLab CI with docker:stable-dind, as it's based off of Alpine. I can imagine this happening all over the place.
Trying to follow the docs as an Alpine user who doesn't know about Rust is kind of confusing right now. As the Alpine specific steps only tell you to install gcc but as you point out above latest Alpine actually has a new enough rust in APK.
The documentation for Alpine builds has been improved #5763 already. I have a pending PR for Fedora, CentOS, and Debian at #5764 . If you have any suggestions for better build instructions, please help by opening a PR.
For myself, I'd like to note that a heads-up in the CHANGELOG / Release Notes (I may have missed it) would've helped:
- when considering the move ("we're considering doing this, please give feedback")
- when it was decided it should actually happen ("release X will optionally contain Rust, release Y will have mandatory Rust")
I don't see any mention before the release in the CHANGELOG or in the emails to python-announce-list.
That's the one thing which distro maintainers or packagers really should be reading on every new release. Including it as a warning in previous releases would've helped us out here to both object/raise concerns and begin preparations.
To be fair the developers did ask for feedback (in a not very inviting way):
We are interested in feedback in things we can do to make this a smooth process, particularly from folks who are re-distributing cryptography. Please note: feedback of the form “don’t use Rust” is not productive and will be disregarded
I highly suspect there will be many more complaints throughout the day about stuff breaking. I had stuff breaking on amazon linux and was confused as to why pip install ansible was asking for rust, seems that one is tagged to latest cryptography in requirements.txt (which is of course not directly the fault of the devs here).
A little piece of information upfront or a deprecation warning would have helped a lot. Or maybe I just overlooked something.
Looks like you have missed all announcements and FAQ entries.
Rust bindings kicked off in July 2020 with #5357. Alex started a thread on the cryptography developer mailing list in December to get feedback from packagers and users. The FAQ also contains instrutions how you can disable Rust bindings.
I suppose this is perhaps a bit naive. I don't blame you, I'm not sure what else you could've done to inform your users either. That said, you need to realize that your project has just introduced itself to a few thousands developers by killing their CICD runs. Talk about first impressions 😄. That's a good reminder to update the alpine images though.
Hands up: who here watches every update of every transitive dependency in each app/environment they have to maintain?
BTW. This actually gives me ideas about dependency change notification systems for developers. Notifying users about incompatible dependency changes has been an issue in this industry for what, last 25 years? How much has changed wrt this during that time?
A little piece of information upfront or a deprecation warning would have helped a lot. Or maybe I just overlooked something.
Looks like you have missed all announcements and FAQ entries.
I suppose this is perhaps a bit naive. I don't blame you, I'm not sure what else you could've done to inform your users either. That said, you need to realize that your project has just introduced itself to a few thousands developers by killing their CICD runs. Talk about reminders to update to the newest alpine 😄.
I totally agree. I really think we did not even scratch the tip of the iceberg. There might be some "interesting" problems ahead.
And don't get me wrong @tiran, I also agree with you regarding your security concerns, but this switch could have really done some serious damage. We will see...
Only users on Alpine (musl), BSD, other hardware platforms, and distro packagers are affected.
other hardware platforms
lol
It sounds like y'all are getting an unexpected lesson about the flip-side of open source software. I would like to push back on some of the entitlement I'm sensing reading in this thread.
Before I begin, I'd like to remind you that security is a numbers game. If the cryptography maintainers can help 90% of their users by switching to a modern memory-safe language, then they'd be irresponsible holding back just because among the remaining 10% there exist fringe platforms which can't even run a Rust compiler.
You can't really expect volunteers to port the whole Rust to other architectures just to get cryptography back.
It sounds like you're saying that your time as a Gentoo and NetBSD maintainer is worth more than the time of cryptography maintainers. They are volunteers as well, y' know. You expect those volunteers to keep their security-focused project dependent on inherently insecure technology because it would make your own job easier. Your goals and requirements might not be matching the goals and plans of the maintainers of cryptography. It might be unfortunate for you but it really is as simple as that.
In fact, your only real recourse long-term is exactly to band up volunteers to make Rust work on your platform. Or pay some hired guns to do the initial work for you. After all, you will be seeing more and more projects depending on this programming language as time goes on. Yelling at creators of those projects for choosing a language that is inconvenient for you is both ineffective and laughable.
Do you seriously expect people who maintain a few hundred Python packages to follow development mailing lists of every single one of them?
It's a security-focused package. If you are not already paying attention to its development then you're putting your users at risk.
I guess this means we'll have to fork Cryptography, and render the original package irrelevant.
I invite you to do that. Please put your money and time where your mouth is. Report back in a year's time how it went.
Let's depend on Rust because everything does!
Let's depend on glibc because everything does!
Let's depend on GNU coreutils because everything does!
Let's depend on systemd because everything does!
Let's depend on Linux because everything does!
Afternote: should be "HARD depend" - without the ability to toggle building with the given component.
We have been able to fix our alpine Pipelines by following the suggested changes in the documentation but they are now extremely slow. We have gone from 30s to 4min. The build of the cryptography package that used to take a few seconds now takes minutes.
We have updated alpine, installed all the dependencies listed in the docs: sudo apk add gcc musl-dev python3-dev libffi-dev openssl-dev cargo and we are using Rust>1.45.0.
Any idea on why the compilation with Rust is that slow??
How could we speed it up??
Building wheels for collected packages: cryptography
Building wheel for cryptography (PEP 517): started
Building wheel for cryptography (PEP 517): still running...
Building wheel for cryptography (PEP 517): still running...
Building wheel for cryptography (PEP 517): finished with status 'done'
It's really not just "some other platforms". Ubuntu 18 is also affected... We are using system distribution of Python to run some automation on Ubuntu images built for AWS and Azure.
cryptography is a transitive dependency in our code: through jira, which requires oauthlib. Of course, in part, the blame is on pip, which makes it very hard to signal that the new version of the package is incompatible / wouldn't try to find the actually compatible version. But you, as a maintainer, also did a very bad job at signaling this to pip.
If you really wanted this breaking change, the only real way to do this as of right now (wrt' tools like pip) is to create a package with a different name. Changing versions won't really help due to transitive dependencies other package maintainers may have no control over.
I'd be totally fine with there being a cryptography2, with newest bells and whistles. But, I have zero incentives for updating the dependencies for our automation tools. It's already too damn hard to find a combination that works. The "benefits" of Rust being a "more secure language" are completely worthless in my case.
@wvxvw creating a new package would actually make the situation a lot worse. Except if the Python module name is also renamed, which would require everyone to adjust there code. Which would also be worse IMO.
It's really not just "some other platforms". Ubuntu 18 is also affected... We are using system distribution of Python to run some automation on Ubuntu images built for AWS and Azure.
Ubuntu Bionic LTS has recent Rust and Cargo version. pip is the problem here. Ubuntu 18.04 comes with an old, unsupported pip version. I recommend that you install a newer pip version in a virtual env. It might also be worth the trouble to file an update request with your vendor.
# echo $(source /etc/os-release; echo $VERSION)
18.04.5 LTS (Bionic Beaver)
# apt update
# apt install -y rustc python3-pip
# rustc -V
rustc 1.47.0
# pip3 -V
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)
@ambv I'd agree with everything you've said, if it wasn't for this line:
It's a security-focused package. If you are not already paying attention to its development then you're putting your users at risk.
With all due respect, it seems to me like this is a strawman. We're not talking about a security update here, are we?
I think (though I'm not sure now), that we have a reasonable expectation in this industry, that such changes will be covered by the versioning scheme.
Currently the lockfile in one of repositories I maintain shows 354 dependencies. The main dependency list includes around 50. The app that is broken for me is a staging environment for a distributed hardware testing platform. This is as far from security concerns as you could get, there's no way I'm able to inspect every single transitive dependency for updates, and there's even less probability that management permits me to spend my time on it.
Please have some empathy. I'm really not blaming any cryptography maintainer, I don't have the time and energy to - I'm fighting an NP-complete problem and this update is making my life significantly harder currently.
It sounds like you're saying that your time as a Gentoo and NetBSD maintainer
We have been able to fix our alpine Pipelines [...] but they are now extremely slow. We have gone from 30s to 4min
It's really not just "some other platforms". Ubuntu 18 is also affected...
Also, now It's not just about Gentoo, is it? Don't you think the 90% figure is a bit exaggerated?
@felixfontein absolutely false... I'd be able to live forever with cryptography instead of cryptography2. And, in case I really wanted the update, I'd be able to update. pip integration is the major problem here, as I just wrote. There's no good way to tell pip that new version of the package is incompatible, beside renaming the package. I believe that the change made to cryptography warrants such a move.
@tiran Sorry, not going to happen. I'm neither updating pip, nor am I installing Rust in those images. Our product is also written in Rust, and your dependency on Rust is completely unnecessary for us to consider in our CI. pip is the version it is for a lot of reasons too.
For me, a solution that would involve embedding the entire jira package with its transitive dependencies in our own deployment would be easier than trying to deal with your demands to our CI environment...
@wvxvw have you ever dealt with cases where pip packages were renamed, but the corresponding Python modules weren't? If you didn't, believe me: it's not fun.
@tiran Just my two cents here.
I would say, that any improvement that reduces the risk of security vulnerabilities is a good improvement. If rust really improves the memory management and helps avoids some future vulnerabilities-go for it. But there are only two things that concern me:
-
Is rust itself stable and secure enough? May there be that there is/will be an vulnerability in rust, which will be impossible to fix inside the library code itself?
-
And the main thing. There is nothing wrong with improving the code or making backwards-incompatible changes. But I think that what should have been done is a reverse of what it is done.
Maybe instead of making rust required in this release, and making it optional by setting an env variable, it could be made optional in this release, that could be enabled by an env variable, and a notice to the changelog could be added:"Added an improved version of the library using Rust. Please help us test that by setting
CRYPTOGRAPHY_USE_RUST=1at the installation step. It will be required in the future release."
That way, nothing would break, but everyone would have been warned and way more people could adapt their workflows and help test the new implementation.
Of course, probably nothing can be done right now after the release is done. Or maybe could it be implemented that way?
I am also using cryptography as a non-direct dependency in docker image builds (alpine linux, via electrum)
@felixfontein correct me if I'm wrong, concurrent.futures and pathlib would be the examples, maybe urllib too.
Of course, it's not nice, but it's a problem that is fixable entirely on my side. I cannot fix the dependency on cryptography for others though, not on my side at least, w/o completely taking over another package.
@wvxvw for pathlib / pathlib2, the name of the Python module changed (from pathlib to pathlib2). In that case, a rename makes sense, because the semantics change. (I haven't looked at the others. I guess they will have similar reasons.) In any cas,e that's not the case for cryptography, using the library does not change (except possible removals of deprecated features). Just renaming everything because a new dependency was added is pretty extreme and very unfriendly to most users of this library.
Different topic: doesn't a simple constraints file with cryptography < 3.4 help in your case?
👋 good morning everybody. This ticket contains a lot of different complaints about the 3.4 release, which incorporates Rust. First: I'm sorry this transition has not been as smooth as we'd all like.
Some context: For several years, @reaperhulk and I have been doing research and publishing on the security threat posed by memory unsafe languages like C. Just last week I spoke at Enigma on this. We believe long term adoption of memory safe languages is nothing short of a moral imperative.
I'm not going to rehash everything that's been said in this thread, but to summarize my views: We made substantial efforts to communicate this change in advance, it's possible to disable this change with a simple environment variable (for this release only), we invested a lot of our time and energy is improving the Rust integration with python packaging (including contributing PEP384 support to pyo3).
And finally: No, I don't care that Rust is not well supported on HP-UX, AIX, and similar. You'll notice I'm being cavalier and not at all empathetic on this particular point. That's because these are incredibly minority platforms, and we (speaking more broadly) can't hold up security improvements for platforms stuck in the past.
On a philosophical note, I'm surprised to discover that so few users of Python packaging employ lockfiles/exact version pinning.
With all that said, let's get constructive!
I think there's a few categories of issues going on here, let's talk through them -- I'll sketch out my views and would appreciate feedback:
-
There's lots of users who could be consuming wheels (in which case they have no need for Rust to be installed), but for the fact that their pip is out of date. Here, I'm not sure there's anything to be done but encourage people to update their pips -- we already do this in every piece of documentation we have.
-
Users with pips that predate PEP517 get an
ImportError. This is another case where updating pip resolves it, and we've made a change to have this print out additional debug information. -
Users with older versions of rust installed experience nasty compilation errors. Here I think we can a) improve our docs to clearly indicate the version of rust required, b) more clearly suggest installing rust with the docs from upstream, c) think about how we might implement the version check directly and thereby give a clear error message (rust-lang/rust#65262 (comment) would be best, but it's not yet stable)
-
It's clear that a lot of alpine users would benefit from having wheels. Someone (not me!) ought to do the work to write a PEP for an alpine build tag, then we'll ship wheels for it and reduce the burden on all those users.
-
It's clear that, despite our best efforts to communicate this, folks were taken by surprise. Where would you have liked us to post this circa 6 months ago?
and are now forced to upgrade all their containers, ci/cd pipelines, packages and so on
I see how this may make you feel frustrated I don't think anybody is forcing you to do anything. It's completely your choice like you chose to use the library out of your own free will (instead of implementing your own, I guess). Blaming things on others is rather destructive thing to do. Besides, did you pay the maintainers for any of the great work they've been doing all these years? Did you pay them for not introducing a safer technology? This comes off as thankless.
I think that it is clear that you have broken semantic versioning with both the python 2 change, and the requirement for rust at build time.
Nobody promised semver and it's publicly documented at https://cryptography.io/en/latest/api-stability.html.
BTW. This actually gives me ideas about dependency change notification systems for developers. Notifying users about incompatible dependency changes has been an issue in this industry for what, last 25 years? How much has changed wrt this during that time?
I think a combo of pinned envs + dependabot (or similar) + CI can catch this just fine. Pinned envs are reproducible, and any incompatibilities are clearly visible in automated bot bumps when they explode in CI w/o automatically breaking the main branch.
We have been able to fix our
alpinePipelines by following the suggested changes in the documentation but they are now extremely slow. We have gone from30sto4min. [...]Any idea on why the compilation with Rust is that slow??
How could we speed it up??
How about:
- Cache base docker layers/images (as in all the static deps are built ones and so install+compile is not invoked in runtime)
- Cache pip wheels — pip builds wheels out of sdists when they are installed for the first time, any subsequent installs that select the same version will pick up those wheels and just unpack them (platform-specific wheels are zip archives with pre-compiled C-extensions)
[...] Of course, in part, the blame is on
pip, which makes it very hard to signal that the new version of the package is incompatible / wouldn't try to find the actually compatible version.
Hey, let's not go into assigning blame here. Plus there's absolutely no fault of pip here — it resolves the deps correctly per declared metadata. Besides, only people involved in the development can dictate what should be done and how, consumers who just come in to do the blame and expect to benefit from free labor are in no position to demand how the volunteer efforts are being spent. The maintainers don't owe you anything — by using this software you agreed with its license that says that it's provided "as is" (the license files are in the repo, I encourage you to read them). Also, you are free to fork the project and set your own rules there (for as long as they comply with the licenses, of course).
But you, as a maintainer, also did a very bad job at signaling this to
pip.
This is not true, the maintainers did a good job within what the current state of the Python packaging allows. I do agree, though, that the packaging ecosystem could be improved. But again, it's run by volunteers meaning that you could influence it by making a volunteer effort. It would be a good idea to even just participate in discussions around improving the distribution package metadata and contribute your use-cases and ideas. Those discussions can be found at https://discuss.python.org/c/packaging/14.
Some topics for improvements are tags for Alpine, as @alex rightfully mentioned, and less widespread things like BSD and others. I believe this would require somebody working on a PEP similar to ones introducing manylinux.
If you really wanted this breaking change, the only real way to do this as of right now (wrt' tools like
pip) is to create a package with a different name. Changing versions won't really help due to transitive dependencies other package maintainers may have no control over.I'd be totally fine with there being a
cryptography2, with newest bells and whistles. But, I have zero incentives for updating the dependencies for our automation tools. It's already too damn hard to find a combination that works. The "benefits" of Rust being a "more secure language" are completely worthless in my case.
It would be a good suggestion at the time when the maintainers were collecting the feedback, not post-mortem.
I think (though I'm not sure now), that we have a reasonable expectation in this industry, that such changes will be covered by the versioning scheme.
It is covered, see: https://cryptography.io/en/latest/api-stability.html. In particular, it says:
X.Yis a decimal number that is incremented for potentially-backwards-incompatible releases.
Our product is also written in Rust, and your dependency on Rust is completely unnecessary for us to consider in our CI.
pipis the version it is for a lot of reasons too.
You seem to expect deprecated software to be supported virtually forever. I believe it's absolutely possible, you'll just need to pay somebody to perform such a maintanence.
@felixfontein correct me if I'm wrong,
concurrent.futuresandpathlibwould be the examples, maybeurllibtoo.
I believe Felix is talking about pypa/pip#8509 — TL;DR because pip doesn't yet have a concept of transactions, files claimed by two different packages get wiped out on upgrade.
Only users on Alpine (musl), BSD, other hardware platforms, and distro packagers are affected.
Presumably I'm doing something wrong, but this hit me on Debian 10
Presumably I'm doing something wrong, but this hit me on Debian 10
Sounds like you've hit an old version of pip as mentioned before. Be aware that deb-like OSs have policies in place that hugely change the whole Python experience in those envs: https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e.
@alex I've created https://discuss.python.org/t/wheels-for-musl-alpine/7084 for Alpine.
Where would you have liked us to post this circa 6 months ago?
There's no one place that could've fulfilled this purpose. To me the issue is that we as a community don't have standardized channels for such communication. I think this issue is bigger than the project in question.
It is covered, see: https://cryptography.io/en/latest/api-stability.html.
Yup. And that scheme failed us today, when lots of maintainers realized that some of the dependencies they use are made with total disregard for this. People assume it's semantic. We either start acknowledging that fact, or we'll be introducing a major breakage in the ecosystem every other quarter.
Assumptions are the mother of all f*ck ups
Yup. And that scheme failed us today
[...]
Assumptions are the mother of all f*ck ups
TBH there are a lot of packages in Python ecosystem that use this scheme starting with ansible-core, ending all the packages that use CalVer, for example. Going further, among OSs that many in this thread use, there's Ubuntu that does CalVer too. So it's not like there's an assumption in the community that every project would use SemVer automatically. Reading the docs about the versioning expectations is on those who specify the dependencies in their envs, many libs don't have this properly documented but this one does and it is very clear about what to expect...
There's no one place that could've fulfilled this purpose. To me the issue is that we as a community don't have standardized channels for such communication. I think this issue is bigger than the project in question.
The docs at https://cryptography.io/en/latest/community.html point at an ML and an IRC channel. This should be enough honestly. More places make it harder to track things... OTOH I'd probably encourage the maintainers to enable Discussions in the repo settings as this provides a great platform for Q&A style posts.
So it's not like there's an assumption in the community that every project would use SemVer automatically
I'm not talking about end-user apps and huge behemoths such as Ubuntu. I'm talking packaged APIs. I haven't checked, but I think that if I do, it will turn out that the vast majority of packages on PyPI use SemVer. Or any scheme that uses "if the first digit changes, you'll have to open your IDE, sorry".
More places make it harder to track things...
No, not forums to discuss it - a functional scheme (e.g. in PyPI), that would allow a maintainer to notify their users about a breaking change in advance. We all do this using a weird amalgamation of emails, forum posts, pinned issues and warning messages. Why?
The signal to noise ratio for this is so small that it becomes tedious to stay alert.
I don't want to subscribe to every possible message on the topic there is, I'd like to have something that gives me a clear signal that says "dude, if you don't fix your crap in 1 week, it will end up in flames".
FWIW, our version/API compatibility scheme comes from Django.
A PyPI feature to let us pro-actively notify our users seems like a good idea, I'd encourage folks to discuss that on the PyPI issue tracker!
I did never blame anybody. I just said, that this change is forcing developers to adapt their CID/CDs a.s.o. and asked to take back this change. Nothing else....
The problem is, that this great library is used as a dependency in a lot of other packages and I guess many users are not aware of that. So just using another library won't fix anything - at least not for me - because that would mean that I also would need to change many others.
In fact I am very thankful for all the work, but I think open source does not mean, that I can't speak about my concerns!
So please read my comments or try to understand them instead of accusing my of being not thankful!!
FWIW, our version/API compatibility scheme comes from Django.
Which, honestly, might not be a good idea. Django assumes that it runs your code (i.e. inversion of control), not the other way around, so breaking changes are unlikely to have that much of a fallout there.
Simply put, Django, Ubuntu, Chrome, Pip, & others that use non-semantic versioning are almost never a transitive dependency. That's why using CalVer there doesn't come with a high cost for people that use it. In the case of Cryptography however...
No, not forums to discuss it - a functional scheme (e.g. in PyPI), that would allow a maintainer to notify their users about a breaking change in advance.
This is something that should go to https://github.com/pypa/warehouse or https://discuss.python.org to see if people would want to discuss implementing it.
We all do this using a weird amalgamation of emails, forum posts, pinned issues and warning messages. Why?
The signal to noise ratio for this is so small that it becomes tedious to stay alert.
I don't want to subscribe to every possible message on the topic there is, I'd like to have something that gives me a clear signal that says "dude, if you don't fix your crap in 1 week, it will end up in flames".
I think that the deprecation warnings serve this purpose rather well. But in this case, instead of a deprecation warning that would be happening in runtime, there was a build-time error that I'd consider a sort of a warning too: there's no real functional Rust code on the main branch right now — it only has a no-op with the main purpose of battle-testing the build toolchain setup at https://github.com/pyca/cryptography/blob/master/src/rust/src/lib.rs. I believe that this can be considered a "soft bump" of the build deps, it's easy to disable for one version with an env var, as described in the FAQ. It seems to be way smoother than merging lots of Rust and using it right away. The mechanism with the env vars is almost the same as with any previous deprecation.
In fact I am very thankful for all the work, but I think open source does not mean, that I can't speak about my concerns!
So please read my comments or try to understand them instead of accusing my of being not thankful!!
It's good that you clarify this now: the problem is that in the text medium many language constructs can carry some sort of emotion and we need to care not only about what we write but also how we do it because it's somebody else who is going to experience the reading and can perceive the same words differently. Especially with the surrounding context saying that they "did a very bad job", or "in part, the blame is on pip" — this explicitly says "blame", I didn't interpret it in an arbitrary, unless there's another meaning of the word that I don't know...
FIRE IN THE HOLE
I think that the deprecation warnings serve this purpose rather well.
You could've put a large ASCII art of a kitty in the warning message and I still wouldn't notice. The reason we have automated pipelines is so that we don't have to look at them until they give us a signal.
Don't think I won't start including giant ASCII doggos...
I agree wholeheartedly about the merits of automation, and it's clear there's gaps here where there's no infrastructure available for some of that.
That said, I think the folks most fully embracing automation (pinned dependencies + dependabot for updates) would have experienced this relatively smoothly.
It's a security-focused package. If you are not already paying attention to its development then you're putting your users at risk.
With all due respect, it seems to me like this is a strawman. We're not talking about a security update here, are we?
@MrMino, the OP here is a Gentoo package maintainer, not an end user. You'd expect repackaging security-related libraries is done with more care than a simple fire-and-forget script in CI.
I think (though I'm not sure now), that we have a reasonable expectation in this industry, that such changes will be covered by the versioning scheme.
While this particular project's versioning scheme is different, I agree that a 4.0 version number would signal the dependency change in a clearer way. Maybe they'll make 3.5 into a 4.0 because there you won't have a magic "get out of jail free" environment variable.
Ultimately though the issue we're commenting on would still be created by someone and we would still be having the same discussion. The developers did communicate their change through their available channels and it was missed/ignored. And most people have this dependency as a transitive dependency.
Currently the lockfile in one of repositories I maintain shows 354 dependencies. The main dependency list includes around 50. The app that is broken for me is a staging environment for a distributed hardware testing platform. This is as far from security concerns as you could get, there's no way I'm able to inspect every single transitive dependency for updates, and there's even less probability that management permits me to spend my time on it.
If Nokia's management is fine running random code from the Internet on your infrastructure then that's your call but ultimately large organizations ought to do some sort of dependency gatekeeping. I can tell you this from first-hand experience as this used to be one of my responsibilities. You can find other FAANG employees telling you the same.
In any case guilting unpaid volunteers by mentioning their actions will inconvenience your paid job isn't a good view. If the maintainers are unable to maintain an LTS version with C code alongside the current version based on Rust, nothing you can say will change that.
Also, now it's not just about Gentoo, is it? Don't you think the 90% figure is a bit exaggerated?
Note that Rust is available on Gentoo, just not on fringe architectures like alpha, hppa, ia64, m68k, and s390. None of those are even actively supported by CPython. s390x and ppc64le are "best effort" and at least we have stable buildbots for them.
90% was obviously a rhetorical figure but we can roughly estimate it for instance by looking at the PyPI download stats for the most popular dependency for Python. On February 6th, it has seen 3.17 million downloads on Linux. That's 96% of the downloads. So Alex's claim that non-Linux platforms are "incredibly minority platforms" is well supported by those numbers, especially if you take into account that another 2.5% is Windows.
We can also look at Python version proportions and we'll see that on that same day 60% of downloads were from Pythons 3.7 - 3.9. That means they're newer than Ubuntu 18.4 LTS which ships with Python 3.6, which also has been established can be fixed by upgrading pip. In fact, 19.9% downloads were from EOL versions of Python, including Python 2.7 - 3.5 which are irrelevant in this discussion because cryptography 3.4 does not run on any of them. So excluding those from the proportions, roughly 75% of global downloads are from Linux running Python 3.7+.
It would be cool to see those downloads per distro, per CPU architecture, or per pip version but it is what it is. So yeah, somewhere between 75% through 95%. We'll see this number move up as Debian 11 gets released this year, and so on. For others, upgrading pip or using rustup is a one-time change that the user needs to make to carry on. This includes Gentoo, Ubuntu 18.4 LTS, and Alpine Linux. Unless you're on m68k...
Finally, this is somewhat off-topic but Alpine Linux itself is a weird choice for a Python container given that it's slow and CPython does not run any stable buildbots on either Alpine or any other musl-based distro.
Note that Rust is available on Gentoo, just not on fringe architectures like alpha, hppa, ia64, m68k, and s390. None of those are even actively supported by CPython. s390x and ppc64le are "best effort" and at least we have stable buildbots for them.
Nit pick: s390 and s390x are different platforms. According to Wikipedia, s390 was a mainframe system released in 1990 and discontinued in 1998.
Cryptography's Rust extension builds fine on s390x. https://koji.fedoraproject.org/koji/buildinfo?buildID=1704720
@ambv @alex Please note that this was not mentioned in the CHANGELOG or release announcement emails beforehand as far as I can tell (I may have missed it, but if I did, that might mean shouting louder would've been a good idea -- i.e. more than once) which is what would've been needed to ensure any sensible packagers or consumers are picking it up.
I don't think it's doable to expect people to read the entirety of development mailing list archives to find what is needed...? I try to do this sometimes but it's not feasible to do it all of the time. Anything critical or important should be mentioned in release notes, and in this case, IMO ahead of time.
Keep in mind that some of us maintain hundreds of packages and I think we give better care to things by not becoming entirely consumed with the minutiae of development of each of them. In this case, yes, it would've been great to see, but an RFC on a random patch? Probably not. Other people are way better placed to give their views on those.
You might also say we should read all commits (or at least summaries) and I do actually try to do this and review diffs to ensure they look sane, but I'm not sure that everybody can or does do this in reality.
By "beforehand" do you mean "at the time of the 3.3" release? Because that's correct, it has been included in the changelog for master for a while though.
My version is pinned down to 3.3.1 and still facing the problem over alpine image.
What's the point of pinning down a version if I suffer from such upgrades?
cryptography==3.3.1
@alex Right, that's what I meant. It would've helped me out (and some others out, based on the upvotes) at least to know beforehand before any Rust was written so I could share any views or useful perspectives, but also a "get your engines ready" a release or two in advance.
I understand in this case that I wouldn't have been able to dissuade you from using Rust (and I like Rust!) but I could've possibly raised notes re other platforms, and that may have influenced when Rust was e.g. made mandatory. I don't know - that's just speculation, I'm just saying that opportunity to share views at a critical juncture would've been neat.
Separately, a "hey guys, stuff is going to break in a few releases" would've definitely been helpful to give a kick-up-the-backside to get some bootstrap binaries built and other work needed to package Rust where we have non-Tier 1 Rust support.
I try to keep an eye on development in packages I maintain or have an interest in, but it's not always doable. Especially because some packages don't update the CHANGELOG until the time of release (this isn't your fault at all, I just mean it's not like a workflow I can universally apply).
There's enough to do with the changelogs, deluge of emails daily, and release notes of packages which need updating and bugs fixing without having to go searching for issues which may not exist. But yes, obviously if it's a core/critical/security-relevant package, I look harder.
the OP here is a Gentoo package maintainer, not an end user.
Forgive me if I misread your comment, but I'm pretty sure that you haven't specified who are you responding to. Many other users are now piggybacking this thread to moan about their particular cases, so I assumed you're responding to all of us.
If Nokia's management is fine running random code from the Internet on your infrastructure then that's your call
Sorry, no. You're reframing my comment into something I have not said, and putting me into a very uncomfortable position by mentioning my employer. I don't like this tactic at all.
- This is not about security.
- This is not random code.
- We don't run random code on random infrastructure.
- The update has not been flagged by anything that would alert me to it.
- Running random code is what staging environments are for.
- I'm responsible for staging environments.
In any case guilting unpaid volunteers by mentioning their actions will inconvenience your paid job isn't a good view.
Just so we're clear: I'm not. And I disapprove of anyone who's doing that.
I'm rather grumpy about the fact that it had to happen and I try to find a constructive way out of it, but that's it. I am incredibly grateful for all the work the maintainers have put into this project. If any of my comments read as if it's otherwise - tell me which one, I'll edit or remove it.
You could've put a large ASCII art of a kitty in the warning message and I still wouldn't notice. The reason we have automated pipelines is so that we don't have to look at them until they give us a signal.
They give you a signal. But by default, it's ignored. The best practice is to use python -W error in CI and filterwarnings = error in pytest that will notify you about such things immediately (this allows you to ignore specific warnings temporarily until you fix your compatibility) — the first entry in FAQ mentions this (https://cryptography.io/en/latest/faq.html#i-cannot-suppress-the-deprecation-warning-that-cryptography-emits-on-import) and here's pytest docs showcasing the use of it: https://docs.pytest.org/en/stable/reference.html#confval-filterwarnings
was no statement from me. It was from @wvxvw ;-) just to clarify...
I'm sorry, I must've confused the quotes with this much hate happening here. Still, your comment about being forced to do things can also be interpreted this way.
@omersi your problem doesn't look related to this discussion. But I guess you're seeing the need to compile things because Alpine doesn't match manylinux tags (that's expected) and there's no standard for pre-compiled Alpine wheels (see https://discuss.python.org/t/wheels-for-musl-alpine/7084).
@webknjaz Nice. I'll need to read up on that and I'll make sure that the pipelines do alert me in the future then. Thanks.
Adding to the CHANGELOG in advance of future releases is a good idea, and one we'll try to incorporate.
@alex Thanks! A shout out in the stuff beforehand so I know when I'm doing a release to keep something in mind for future is a big help.
Actually, I love such updates because they usually move the industry forward, showing the need in new features. I would love to see alpine linux wheels standardized, but actually this issue and some pieces of advice not to use alpine have urged me to compare docker images (a little bit offtopic, but still).
My current alpine image is using lots of compiled packages, including cryptography (even before that release it was quite a lot, but now with rust leftovers it will probably be even more in size).
I have tried to do the same with an python:3.6-slim-buster image instead of python3.6:3.6-alpine.
And to my surprise, the end image was 232 MB instead of 315 MB (26 % less) of the alpine image, plus that image uses wheels (so such changes like this would do nothing), which speeds up the build like 10 times, and makes it possible to even add caching.
So, what can I say, maybe this update is an call for action to check your apps and upgrade to something better (:
First of all, I find it really sad (or even distressing) that my original argument that Rust doesn't support modern Linux systems on minor architectures immediately got twisted into 'HP-UX, AIX, and similar'. No, this is really about up-to-date Linux systems. I can't say we managed to provide superior support to them so far but we haven't been put into the position of having to discontinue their support entirely either.
Secondly, I filed this bug because I thought that maybe upstream hadn't realized how non-portable Rust is and how much problems this causes. However, reading this thread -- and I'm sorry if I'm wrong -- I get the feeling that upstream just doesn't care about all their users. In fact, I'm myself not interested in optimizing stuff for everyone. However, I don't believe this should go at a price of providing zero support for minorities. Slow, pure Python fallback would be at least a nice thing for those who can't use Rust extensions.
Finally, and this personal -- I feel really betrayed here. So far I was advocating cryptography as a really decent replacement for the dead and buried pycrypto library. And now it turns out that cryptography is no longer a replacement for it because it stops working on some platforms we do actively support.
We have been able to fix our
alpinePipelines by following the suggested changes in the documentation but they are now extremely slow. We have gone from30sto4min. The build of thecryptographypackage that used to take a few seconds now takes minutes.
Have a look at https://pythonspeed.com/articles/docker-cache-pip-downloads/
It is now possible to share pip caches between builds which includes the wheeling of binary extensions. This should improve your build times even beyond the 30s.
I really don't think Alpha, s390, or hppa are any less obscure or a minority than AIX or HPUX. But in any event, feel free to substitute them in my comment, I stand by it.
While I'm sorry this is causing difficulties for you, I do not see how this was avoidable. Our users (indeed, all users) are on x86(-64) and ARM. We simply cannot abandon forward progress to platforms that haven't released in decades.
@webknjaz
It started failing today with rust exception.
What do you think caused it?
Alpine 3.10
cryptography==3.3.1
FTR, if anyone from HP, IBM, or DEC cares about this problem, HMU. Happy to work with you to find ways to fund LLVM and Rust.
It started failing today with
rustexception.
What do you think caused it?
No idea, you didn't provide enough info to deduct that just from a few sentences w/o any details. But it is 100% certain that the version you're linking does not have any Rust references at all. I would guess that you may be installing a newer version of cryptography in some other place of your project without realizing it. FWIW this is not the right place to get support.
Alternatively helping on the rust-gcc front-end might help solving at least partially the problem.
I was impacted in my CI running in an alpine container on an ansible collection repo.
I added rust and cargo in my alpine install deps (apk add cargo rust) and got my CI back.
Not a big deal on my side.
FTR, if anyone from HP, IBM, or DEC cares about this problem, HMU. Happy to work with you to find ways to fund LLVM and Rust.
DEC went out of business long ago, HP absolutely does not care about their RISC ISA from 1996 that they've officially stopped support for in 2013 (8 years ago), and IBM is content huffing its own mainframe farts consisting of 60 year old COBOL.
Like everyone, pipelines went down because using Alpine for containers..
Folks who are commenting that your pipelines broke: Do you pin your Python dependency versions (pip-tools compile, pipfile, etc.)?
Although we are pinning dependencies, we saw breakage because Cryptography is a dependency of a direct dependency in our requirements.txt file. In particular it is Authlib==0.15.2 which in turn requires cryptography>=3.2,<4. So pinning all your dependencies does not guarantee preventing a problem like this.
My highest respect for all the folks working on this project on their own time, sometimes it is tricky to avoid breaking changes, even if they are well communicated.
Ah, so you pin direct dependencies, but not transitive ones. Is that what other folks are doing?
For me it was similar, ansible depends on cryptography
@alex Right - same here in one of our builds. We'd pinned direct dependencies but not cryptography.
Can I ask why? Is this an intentional design choice, a tooling gap, something else?
In the pip ecosystem, a constraints file can be particularly useful for pinning transitive dependencies: https://pip.pypa.io/en/stable/user_guide/#constraints-files. It allows you to keep your requirements.txt to only include requirements you directly use, but still be able to pin transitive deps. With pip's old resolver, the constraints file will override all transitive deps to use your version; with the new resolver, it will error if your pin is not compatible with a direct dep.
Here's a simple script to generate that constraints file:
#!/usr/bin/env bash
set -euo pipefail
# You can change these constants.
PYTHON_BIN=python3
VIRTUALENV=build-support/.venv
PIP="${VIRTUALENV}/bin/pip"
REQUIREMENTS_FILE=requirements.txt
CONSTRAINTS_FILE=constraints.txt
"${PYTHON_BIN}" -m venv "${VIRTUALENV}"
"${PIP}" install pip --upgrade
"${PIP}" install -r "${REQUIREMENTS_FILE}"
echo "# Generated by <relative_path_to_script.sh> on $(date)" > "${CONSTRAINTS_FILE}"
"${PIP}" freeze --all >> "${CONSTRAINTS_FILE}"Note that you don't need to include all your transitive deps in a constraints file! You can have a constraints file with only one entry.
@alex I suspect a lot of casual projects like mine (and not-so-casual ones apparently) only pin direct dependencies rather than pip freeze, or Pipenv, etc.
In case it's helpful to anybody, the principal tools for pinning all dependencies are:
pip-compilefrom pip-tools which is easiest if you have an existing setup.py- Poetry which is easiest for new projects.
- (Pipenv does this too but I would not recommend Pipenv.)
The workflow is to have two files: in the "constraints" file, you define your dependency constraints manually. Then use a locking tool like pip-compile or poetry lock to generate a separate "lockfile" which contains all of the pinned transitive dependencies and hashes.
@alex I suspect a lot of casual projects like mine (and not-so-casual ones apparently) only pin direct dependencies rather than
pip freeze, or Pipenv, etc.
There are likely large projects out there which do not - OpenStack being one example. Vendors whom package into their own python packages almost certainly don't fall into this category. However, if you pull an OpenStack package like keystone from pypi right now, it is part of the ripple effect here.
https://github.com/openstack/keystone/blob/master/requirements.txt#L9
I second the advices above: I recommend using pip-tools, it's the most native thing that pip can use directly. Other lockfile formats are usually specific to custom package managers but should also achieve the same goal.
If your project is on GitHub, I recommend enabling Dependabot — it supports pip-tools managed constraint pairs, so when it sends PRs updating separate dependencies, each such change gets tested and it's quite transparent when something in the dep tree breaks, even if it is transitive — you'll know which one right away.
It started failing today with
rustexception.
What do you think caused it?No idea, you didn't provide enough info to deduct that just from a few sentences w/o any details. But it is 100% certain that the version you're linking does not have any Rust references at all. I would guess that you may be installing a newer version of cryptography in some other place of your project without realizing it. FWIW this is not the right place to get support.
Found the root cause
other libs were using unpinned versions causing this failure.
Bounty for LLVM+clang+Rust on m68k Linux/NetBSD/AmigaOS when?
Can I ask why? Is this an intentional design choice, a tooling gap, something else?
I think it is tooling gap. pip freeze is not that helpful when you want some packages to be pinned but not all.
I spent whole day cleaning up the mess this change caused. I had ansible pinned but it was pulling the latest cryptography.
As people suggested above pinning cryptography as well is the solution.
I think it is tooling gap. pip freeze is not that helpful when you want some packages to be pinned but not all.
pip-tools exists to help you with this exactly. TL;DR you can track your direct deps in a requirements.in file, having loose caps or none at all but then you use pip-compile which generates requirements.txt with the whole tree of transitive (indirect) dependencies and you can re-generate it over time to update the tree to newer pins for the whole thing (instead of pip freeze). Then, you can pip install -r requirements.in -c requirements.txt and this will allow you to have a separate place to track your direct deps along with a separate lockfile with all the pins that help you keep your env reproducible as those pins will be know good versions of everything.
The security benefits are of course appreciated - but not the way that we got to know about them. Cryptography is a transitive dependency of a transitive dependency of dependency of our code. Our 6 small team that does not deal with code in any way related to cryptography should not need to carefully track the changes in packages that deep in the tree. One of our dependencies updated today - and with it updated cryptography. Our builds started failing. We got to learn about this package, and it's deceptive versioning scheme.
You say you know who your users are. You don't. You don't know any of the users that install your package through a package manager. Ubuntu, Debian, Alpine, Fedora, Gentoo, etc. all maintain packages of this library for the architectures you just inadvertently removed support from. You don't know how many of them install this library, and how important it is for them.
You might not have known that Python tools are horrible when it comes to non-python dependency management. Now that I think about it, maybe it's more worth to just use packages of distributions, that would have prevented all of this from happening. I wish I could on this project. But yet, if I did, you wouldn't see me as a user.
You say it's for the better, you say that the old platforms are not worth keeping secure and should just move on - ignoring the point that by removing support for them, you silently left them insecure, unable to get important updates and vulnerable for attacks. You say the extra dependency is alright, since it fixes a lot of potential problems that could arise if continuing to use C. And this might be right. But you know what also deals with a lot of security sensitive stuff and is written in C? Linux. There has been numerous suggestions to rewrite parts of it in Rust, to improve security. But those suggestions were rejected. Why? Because Linux developers knew, that adding Rust might block their users from using Linux. Linux developers don't know all of their users. But they trie to care about all of them, whether they know them, or not. Linux developers care about their users. You don't.
I'm just here for the whining of people who's build systems are using nondeterministic versions and/or unpinned upstream dependencies.
Folks, if you do this, you're going to get what you deserve. Don't blame the maintainers of this software for your own choice to float dependencies. The level of snide and hate here for maintainers is nuts, and is a solid example in the disincentives to quality software present in the OSS community.
Do better!
@virusdave notwithstanding the fact that I agree about the merits of pinning deps, I don't think this is a productive way to engage.
@alex I think it may be a good idea to lock this topic as it just gets a lot of hateposting and all of the pinning improvement advices have been given already...
@MrMino I did read the thread - the advice given here is passing the blame to someone else (in this case - users). Strongly echo what @ignaloidas has pointed out very well - You say you know who your users are. You don't.
I appreciate the forward facing changes and improvements made, but unless the software being built (whether open-source or otherwise) fails for the very problems it aims to solve - I feel its a lose-lose.
Nobody is blaming anyone, we're trying to figure out how to design resillient systems (https://how.complexsystems.fail/) that work for as many use cases as is practicable. Doing this requires understanding the types of workflows which were impacted.
Certainly it's my hope that organizations which are impacted will run blameless post-mortems to attempt to see how their own processes for managing third party dependencies can be improved.
That said, you've provided no details on your problem. You showed up, complained that something didn't work (without even including so much as an error message) and demanded we revert our release. I think you need to seriously rethink how you engage with open source projects, as that conduct will get you nowhere.
If you want to include error messages and instructions for reproducible, we're more than happy to figure out a) how to fix what you're running into, b) how we can improve the situation to make the next person in that set of circumstances have an easier time. But until then, you don't appear to be even attempting to contribute to this discussion constructively.
@alivcor no, the advice was that pinning your deps is a responsible thing to do. And that a blame culture is counterproductive is just common sense that people tend to forget and therefore need to be reminded about.
That said, you've provided no details on your problem. You showed up, complained that something didn't work (without even including so much as an error message) and demanded we revert our release. I think you need to seriously rethink how you engage with open source projects, as that conduct will get you nowhere.
@alex Now that's simply incorrect (and you pointing out my thinking process is actually blaming users to rethink their process!). I have contributed to open source myself and I think I know what it takes. I didn't feel the need of adding an error message here because the evidence is clearly laid out by 100+ users on this thread. I didn't "demand" the release to be rolled back, I said I was fairly confident that is the best resolution here - and I stand by that. Whether you do that or not - it's your choice, you guys are doing the amazing work here - not me.
I think if I would get such reaction from users of my open source software, I'd definitely lean towards THEIR perspective. As @clanzett pointed out here - open source does not mean, that you can't speak your concerns and offer suggestions - In my case and clearly with many other people here, rolling back this breaking change was the best possible resolution to prevent any more people bumping into this.
I think the best thing to do here would be to provide a pure Python implementation of whatever's necessary. After all, isn't one of the points of using Python in the first place to write memory-secure code? If you're really concerned about the performance here, you can make Rust extensions optional, like many other Python projects provide optimized non-Python paths with Python fallbacks.
I think I agree with @mgorny here - probably moving rust and compiled versions to extras would be a good idea.
For anyone who might be using this library and facing an issue because of transitive dependencies (airflow etc) - Use an older version eg; 3.0.0 - that will satisfy the requirements and you won't need to install any extras.
pip install cryptography==3.0Simply re-implementing everything comes with a number of downsides:
- There's no way to implement constant-time code in pure Python securely
- It requires us to implement all the relevant portions of the codebase twice. While I very rarely make appeals to "we are all volunteers", it's simply that case that it's not good use of our limited time to do so.
- It requires us to expand our test matrix further. We already run an incredibly large test matrix, over 50 builds on every commit. We'd need to introduce the cross-product of that matrix by which implementation we are using, or establish what subset of which we could safely ignore, (ironically, this is likely to make us even less likely to add new platforms to our test and release procedure, as this makes the overhead of new platforms higher)
- It introduces a risk of behavioral differences between implementation, as well as increased complexity.
For all the entitled "users" here who believe maintainers of this package should specially cater to their needs read the license:
Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
You're not paying for this product and you can't expect any special support. It seems to me that some of you people think that free and open software gives you the prerogative to expect free and open support as well. Well that's not how things work.
You've had plenty of warnings about this change in a past couple of months as well.
Using this library and being updated about its changes is your responsibility. And if you have issues down the chain with other libraries using this one then move this discussion to their forums.