--allow-all-unverified probably needed
rassie opened this issue · 46 comments
The pull request that introduced --allow-insecure also included --allow-external, --allow-all-external and --no-allow-external, i.e. setting the option for all, none, one or several packages is possible. Setting --allow-insecure for none, one or several packages is possible, but not for all packages. I think there should be some kind of symmetry in these options and lacking --allow-all-insecure could be a backward compatibility issue as well.
This was purposely left out.
The --no-allow-* options are nooped in the upcoming pip 1.5 and their behavior is now the default. However because --allow-unverified (renamed from --allow-insecure to be more accurate) is a security sensitive parameter it was decided to not allow globally allowing it. The external flags are still secure (the files are hosted offsite but are directly linked too from the PyPI pages and the link includes a hash that the downloaded file can be verified with) and removing them is mostly a reliability and privacy concern so opting in to all external links was added.
In this particular case, it is certainly viable to whitelist every package installed from an "insecure" source in pip.conf, the problem is then synchronising these changes to every developer on your team. Since this problem is more common, I've opened a seperate issue for that.
And I'd be happy if you reconsidered your position: after all, it's still possible to allow all unverified packages, it's just become a vastly more tedious task, so in the end there is barely a security win, but a big chunk of morale and time is lost for the people who absolutely have to deal with unverified packages.
I agree with a configuration system that allows per project configuration better, FWIW --allow-unverified/allow-insecure works inside of a requirements.txt
It's true that you can allow all of the unverified packages you use, but it becomes a much more deliberate action when you need to name the ones you want to whitelist it for. Similar to a browser there's no "disable TLS host checking" but there is a "for this host disable it". I'm happy to think it over more though.
FWIW --allow-unverified/allow-insecure works inside of a requirements.txt
I'm not using requirements.txt for now, since I had no strong incentive to do so -- setup.py worked just fine so far. I suppose the incentive will show itself soon, and it will be "everything else stopped working", which would be kind of sad :) But yes, requirements.txt would work, but the general setuptools use case is still cumbersome.
If I may intrude on this ticket, since it is still open - I stumbled across this change today, and would vote for the inclusion of an --allow-all-unverified flag. In lieu of that though, I cannot seem to get --allow-unverified to work inside of my requirements.txt, and there does not yet seem to be documentation showing the correct syntax. Specifically, my requirements.txt contains:
--allow-unverified httplib2==0.7.2
and pip seems to just skip over the line without installed or reporting a problem. Meanwhile, pip install httplib2==0.7.2 --allow-unverified httplib2 works. Is this syntax incorrect, or has the option to use this in requirements.txt been removed, or is this a bug?
requirements.txt
--allow-external httplib2 # You probably need this as well.
--allow-unverified httplib2
httplib2==0.7.2
They are separate lines, the one just allows httplib2 to be unverified wherever it may be installed from.
Aha, thank you
Given that the code that brought --allow-external and --allow-unverified into existence probably broke many peoples build-systems there needs to be a blanket way to make pip act as it did in the previous version. At least until package maintainers can sort out their stuff.
Leaving people scrambling who have no control over how 3rd party packages have put their software on pypi is a very rough choice. I am sure that out there somewhere, some poor sys admin tried to install a python app today, and failed. Whats more they probably have no idea where to look. Pip is way past version 1.0, and this was a backwards incompatible change. I think its only right to have a --act-old-and-broken flag for a version, to ease the pain. Its about 4 lines of code and in version 1.6 you have every right to remove it and have pip pitch a fit about people calling it.
That code existed in pip 1.4 which issued warnings when installing things that would break in 1.5. It was phased in as described in PEP438 (http://www.python.org/dev/peps/pep-0438/). At this point personally I feel like introducing an --act-old-and-broken flag as you mentioned will merely kick the can down the road. People will flip that on in their build scripts, and go on with everything working again and not solve the underlying problem. That they are depending on a grossly insecure means of installing packages. When 1.6 rolls around that is removed and they are back to where they are today, possibly slightly better off, but overall still in the same general situation.
So given that this has been known for months, and has been emitting warnings for months, I'm still -1 on adding a flag, especially one that we expect to remove in one version, thus breaking things for anyone who uses that flag to begin with.
However perhaps @qwcode or @pfmoore have a different opinion on the matter?
The people who would turn it on are not the package maintainers you seek to drive to action. Punishing the users of packages for the maintainers actions has to be considered the worse path.
The alternative, and what I am now doing in production, is to downgrade and pin pip until package maintainers have caught up. I filed bugs with the respective projects that failed for me. Is downgrading better than a switch? I don't know. I don't think so, but I will leave that to you guys.
I don't believe it's punishing anyone to have them enumerate which packages they wish to allow to be unverified instead of allowing a blanket "allow all packages to be unverified". For the hypothetical system administrator they only need to take a few minutes and attempt to install their software a few times in order to discover the complete list of packages they currently depend on which rely on an insecure method of discovery. Once they've discovered the list they need to only issue pip install --allow-unverified foo --allow-unverified bar foo bar instead of the proposed pip install --allow-all-unverified. Once the initial "sort out which dependencies are currently depending on this" is over, this is actually more advantageous as it prevents them from accidentally relying on more externally hosted libraries without realizing it and it gives them a list of packages they can file bug reports against asking for an upload.
If someone really needs to install something right now and can't take the few minutes to sort out which dependencies fail to install under the new rules, then personally I'm OK with them downgrading. But I'm not the sole maintainer so If @qwcode or @pfmoore have a different opinion then I'll defer to that.
So my question, if you already have a list of packages that you need, why can't you simply add an --allow-unverified call for each one of them?
Has anyone (for example, the original poster) contacted any of the projects hosted insecurely to raise a bug on them and/or ask them to provide a securely hosted copy of their package? Do we have any feel for the projects' likelihood of actually fixing this issue?
I know that when PyPI switched the external hosting flags, emails were sent out to projects at that stage (which is a similar scenario). Do we know what the response on that was like? Have projects bought into the improvements in security that we are trying to achieve?
Essentially, I agree that having package users suffer is less than ideal - but that doesn't necessarily mean that pip should revert this change. If we can help the users persuade the projects to move to secure hosting, that would be worth doing.
I know of a few projects that have been contacted, some have switched, some haven't in 3+ months since original contact.
We don't know the response from that because for most people the switch was completely automatic. We do know that there is still around 6k (out of 38k) packages that rely on either --allow-external or --allow-unverified for at least one version. The caveat here is that when I ran those numbers it wasn't that unusual for a project to be completely hosted on PyPI except for one or two versions that just accidentally didn't get uploaded. Additionally it also counts projects who host everything on PyPI, but also have a ==dev requirement that points to github or similar.
I don't believe I've seen anyone vocally refuse to at least move to directly linking their projects with a hash (which would be enough to allow the --allow-external or --allow-all-external flags to take care of them. So far it seems the projects that I've personally seen either haven't been made aware even with the various emails that got sent out and the warning in pip 1.4 or they have had issues raised against them and they have not had any response (some of them the issues were raised months ago).
I suspect that, similarly to Python3 support, some number of these are simply un(der)maintained and the original authors don't care about them and won't be bothering to upload. For the ones who will be uploading and just simply aren't aware of yet I don't believe there is anything else PyPI or pip itself can do to make them aware of it, and it's up to end users to file bug reports. To this aim you'll need to figure out a list of what packages you rely on to file bug reports against, and once you have that list I don't believe it's any more difficult to add a --allow-unverified for each one than it is to issue a global --allow-all-unverified.
What I do believe is easier is to see you have some unverified and just chuck the --allow-all-unverified flag and forget about it until pip 1.6 rolls around when we then remove it. This will benefit people if someone else happens to have the same dependencies as you and filed the bugs and convinced upstream maintainers to release in the time between 1.5.1 and 1.6. However it will not benefit if something you depend on is un(der)maintained or that everyone who depended on it simply tossed the --allow-all-unverified flag in and forgot about it until 1.6.
All of the above being said, there is #1423 which has a few things inside of it but that some were a duplicate of this issue but which i've semi hijacked to represent the fact that right now if something needs --allow-unverified it also needs --allow-external for it. I believe this is needlessly repetitive and in 1.5.1 I plan to make it so that --allow-unverified FOO implies --allow-external FOO.
I'm also going to add this issue to the 1.5.1 milestone. We should decide, one way or the other, if pip will include an --allow-all-unverified option and if it does include one, if we'll remove it in 1.6 or keep it as a permanent option.
I don't see the logic in having --allow-all-external but not --allow-all-unverified, I also don't see the benefit of making it more inconvenient for people to work around the current packaging landscape.
Look at this as a user rather than a dev for a moment. From a developer perspective its easy and the course is clear. For someone following a howto to get started with, say DjangoCMS your arguing that they should go line by line through a requirements file that they don't maintain and don't understand and modify it. Eventually DjangoCMS(who may not have this issue, but you get the idea) will update but for now there is no way a USER can work around the problem, only developers.
Pip is more than a developers tool now, its part of the plumbing for a large variety of software. Its a testimate to how awesome pip is that this is true. I really don't want to come off as tearing you guys down, you have all made a awesome set of tools that I love and use daily. I just think that is decision counter productive, by being too focused at developers.
The logic is basically this, --allow-[all-]external isn't a security risk. If you use it the worst case scenario (compared to without) is that your install fails because an external host goes down. On the other side of things --allow-[all-]unverified is is a security risk. If you use it the worst case scenario is an attacker gets to execute arbitrary code on the end user's machine.
It feels wrong to me to make it any easier for end users, who might not understand the risks, to give them an attractive "make my problems go away" footgun where the worst case failure mode is an arbitrary code execution.
I'm actually somewhat on your side over this - I do think that we should be viewing this from the user's perspective. But on the other hand, I don't think that pip should have to solve everyone else's problems. In this case, the fundamental problem is that packages are not providing any means of verifying the package integrity, and that's a problem with the package, not with pip. So in the first instance the package maintainer should be fixing this, and users should be complaining if they don't. In the case you mention where the user is looking at "a requirements file that they don't maintain" then they can contact the people who do maintain that requirements file, and point out that the dependencies in there are unverified so the requirements file needs to change to add appropriate --allow-unverified flags. The security responsibility is then on whoever maintains the requirements file. Only if neither of those options work should the user have to do anything themselves.
Of course, reporting bugs is often slow and painful, so having a "quick fix" in pip would be convenient for the user. But as @dtufft explains, it gives the user the opportunity to take a security risk that they may not properly understand.
As a regular user of the --no-check-certificate flag in things like curl/wget, I can fully sympathise with the "just let me get things done" viewpoint. But I don't feel sufficiently comfortable that the viewpoint is right, to argue for such a flag in pip.
The user does also have the option of manually downloading the relevant files, and putting them in a local directory which they specify via --find-links as an alternative location to find packages. That location will be considered secure (at least I assume it will - @dstufft can you confirm that this will work?)
Yes it will work, as will alternative indexes including something as simple as tossing files in a directory and using Apache/Nginx's auto index support. The index that we're contacting has to advertise support using a meta tag (which PyPI does) to trigger the external/unverified code path. If the index does not advertise support using that meta tag then it assumes all urls are internal and verifiable because we have no way of knowing if they are or not.
If this is not the case then that's a legit bug.
I am one of those people who have suddenly had all of their deployments break because of this. I have been deploying code continually for the last year (mostly using chef) and have never seen any warnings. I suppose that's because chef hides the output of the commands it runs.
In my situation, adding --allow-external and --allow-unverified to requirements.txt is not a great solution. We have hundreds of software packages which have been written prior to this change and you're saying we need to change the requirements for all of them?
I completely agree that this should be the default behaviour of pip, but I believe an --allow-all-unverified option should exist as an (un)happy medium. Ultimately, the package maintainers should get on top of this.
btw, I can go either on way on this one. I can see both pts of view.
The way I can see this we have:
Me: Believes this flag is a bad idea
Paul: Appears to lean towards not implementing the flag, though more in the middle than myself
Marcus: Could go either way
Additionally I've reached out to a few people with large scale deployments or people with widely used projects who I know were affected by this change. Most of them expressed the fact that a flag would ease some short term pain, but they all pretty much believed that adding the flag would be a bad decision in the long term.
I'm closing this issue as a wontfix. I want to thank everyone for their input and I would like to apologize for the short term pain it has caused. Hopefully in the long run this pain will pay off for a better ecosystem for all!
Even though we disagree, I have to applaud how this conversation was handled. Thanks!
If I may, I have one use case to consider. I want to know which python packages are outdated, but pip list --outdated complains many of them are externally hosted and insecure. Now, I would like to know which packages are out of date, and then I will manually consider the risk of installing them or not.
I admit, most of these packages are not appropiate for pip (like fedup, Fedora Upgrade or wxpython), and I will ignore them.
So, perhaps a --allow-all-unverified only when the packages are not going to be installed?
i would urge reconsideration of this issue. or perhaps another approach altogether, though its probably a bit late for that.
- pip is not just a deployment system it is an essential development tool.
- security is a way of life and people make different choices in how to achieve it.
First, this would be much less of an issue if only deployments were affected, but this is highly cumbersome while developing. When i'm working on code I typically have multiple virtual environments with different versions of python and/or different combinations of packages installed across different machines. While i'm developing I'm using git:// in dependency_links to make things easier and portable around my local internal development environment and to avoid pushing out pre-release code somewhere it could be mistaken as deployable. This is different for releases that are intended to be deployed.
This does nothing but require me, while i'm developing, to spend time poking around at my tools rather than getting to the point of what I am trying to accomplish. It is bad for the development ecosystem.
Second, insecure connections are not always a security risk, again especially when it comes to development. If I don't want to setup SSL for my internal pypi server on my isolated development LAN i should be able to do so. But pip can't tell if I'm "deploying" or "developing" so it demands a string of command line options longer than my arm.
There are valid important use cases that don't necessarily require the default mode to be insecure but certain warrant an option to more easily override the default.
Please reconsider this issue.
What about a pypirc / ENV option for domains (/IPs?) to allow pip to download over http from?
i would urge reconsideration of this issue. or perhaps another approach altogether, though its probably a bit late for that.
- pip is not just a deployment system it is an essential development tool.
- security is a way of life and people make different choices in how to achieve it.
First, this would be much less of an issue if only deployments were affected, but this is highly cumbersome while developing. When i'm working on code I typically have multiple virtual environments with different versions of python and/or different combinations of packages installed across different machines. While i'm developing I'm using git:// in dependency_links to make things easier and portable around my local internal development environment and to avoid pushing out pre-release code somewhere it could be mistaken as deployable. This is different for releases that are intended to be deployed.
I think there's some confusion here. --allow-external and --allow-unverified have nothing to do with dependency links. There is a separate setting for those called --process-dependency-links, however it's currently slated for removal in 1.6.
Generally the solution for development dependency links is a requirements-dev.txt file that has -e's setup to those repositories.
This does nothing but require me, while i'm developing, to spend time poking around at my tools rather than getting to the point of what I am trying to accomplish. It is bad for the development ecosystem.
Second, insecure connections are not always a security risk, again especially when it comes to development. If I don't want to setup SSL for my internal pypi server on my isolated development LAN i should be able to do so. But pip can't tell if I'm "deploying" or "developing" so it demands a string of command line options longer than my arm.
If you have your internal mirror setup on HTTP the most pip will do is display a warning that it's accessing it over HTTP and not HTTPS. SSL has nothing to do with --allow-unverified and --allow-external. Additionally unless your internal mirror has very specifically opted into "api v2" of the /simple/ API, as PyPI has, then pip will also have
There are valid important use cases that don't necessarily require the default mode to be insecure but certain warrant an option to more easily override the default.
Please reconsider this issue.
I think there's some confusion here
using pip 1.5.2:
- there is no deprecation warning for a setup.py using dependency_links.
- the error message for a failed dependency using dependency links is:
Could not find any downloads that satisfy the requirement [...]
Some externally hosted files were ignored (use --allow-external packagename to allow)
adding --allow-external follows up with a similar error saying to add --allow-unverified.
please fix these as they are confusing.
--allow-external has nothing to do with dependency links.
i understand but i'm saying that the error message doesn't point to the appropriate source of the problem and causes this confusion.
What package are you installing that includes dependency links?
nothing on pypi.python.org if that's what you're asking.
maintaining internal packages means not using public channels.
make a package yourself and add a dependancy_link to a git repo. it will fail in a way that tells you nothing useful and is actively misleading.
Hmm, I'm confused. I can't think of how you'd get that error message you pasted above outside of PyPI unless your index explicitly opted into version2 of the metadata and didn't properly support metadata 2.0.
What are you using to host your index?
this package depends on other packages on pypi.python.org and uses dependency links to depend on local git repos. i'm not using an internal mirror or index for this package.
Can you reconsider adding a --allow-all-unverified command, or at least reversing it such that there is a --allow-no-unverified instead? Of all the types of software to break semver, a package management tool should be way down at the bottom of the list. It's quite annoying to now have to:
- Upgrade everything to pip 1.5
- Use flags that don't work on older versions of 1.4 in our requirements files
- Have overrides for packages that depend upon things outside of pypi
- Have internal packages that are hosted on github (not a pypi index/mirror)
Yeah, I get the security reasons behind this change, but it's unconscionable for you to make such a breaking change in what should otherwise be a minor release.
Pip doesn't use semver and never has, it uses a phased in deprecation cycle. If anything you were installing requires this flag there were warnings in 1.4 to tell you this change was going to happen. If you want something that will work for 1.4 and 1.5+ you can use --allow-insecure instead of --allow-unverified. The flag was renamed in 1.5 to be more descriptive however the old flag continues to work and will continue to work.
I'm going to +1 this issue even though its currently closed.
It took quite a while to get to a workable packaging, build, and deployment infrastructure based on setup.py and pip. This infrastructure has been quite stable until 1.5.1 with introduction of the --allow-external and --allow-unverified behavior. Basically this behavioral change (regardless of its security advantages) is punitive towards users of pip since maintenance of upstream packages is outside of our control. Without a --allow-all-unverified option my only real choice is to downgrade to an earlier version of pip and delay adoption of wheels/etc and other new pip features (which would be nice to have but I can do without).
If there were a way to specify the --allow-unverified argument for each package as part of setup.py, I would be ok with making changes there. Otherwise, manually maintaining separate requirements.txt files will be a real pain (today we auto-generate a requirements.txt for the whole application using pip freeze after installing all internal packages from sdist form).
Clearly for me, the easiest path forward would be to have a --allow-all-unverified flag and a modern pip. An --act-old-and-broken flag would be fine as well - just being pragmatic here.
+1 to this as well.
We use a PyPI proxy for hosting our internal company packages, as well as caching, and this new "feature" is straight up breaking our builds. Obviously it's possible to add the flags in the requirements.txt, but that makes diffing it against a pip freeze useless, and just adds more maintenance overhead.
The Python maxim, "We're all consenting adults," comes to mind. If I know I need to install a bunch of packages from an "insecure" server, please let me do it. Don't force me into a maintenance nightmare or make me downgrade pip in every build to <1.5.
Why is the proxy opting into the newer stricter urls?
I've exactly the same problem as @shakefu .
We are using pypiserver 1.1.6 for host our internal packages. This is part of our continuous delivery system. And the only way that we found of handle this situation is upload the unverified 3rd parts packages in our own server. But this did not scale so good.
+1 on this.
No idea why it's using stricter Urls. We run localshop. —
Sent from Mailbox for iPhone
On Wed, Apr 16, 2014 at 12:23 PM, Donald Stufft notifications@github.com
wrote:
Why is the proxy opting into the newer stricter urls?
Reply to this email directly or view it on GitHub:
#1268 (comment)
+1, this is especially nasty when you have nested private packages that depend on each other -- so that you end up needing to specify --allow-unverified for all of the upstream dependencies. makes very ordinary build and deploy super brittle and difficult. wish you guys would reconsider.
You can use --trusted-host to whitelist your private server.
That certainly seems like it should work, and might work if all of the packages are already on your local pypi. It doesn't work when you set --trusted-host github.com and your packages have nested git+ssh dependencies.