pip 9.0.2 KeyError: 'pip._vendor.urllib3.contrib'
vhaldemar opened this issue · 17 comments
- Pip version: 9.0.2
- Python version: 2.7.3, 2.7.13
- Operating system: Ubuntu 12.04, Ubuntu 16.04
In [1]: import urllib3.contrib
In [2]: import pip.pep425tags
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-2-9ae721442c45> in <module>()
----> 1 import pip.pep425tags
/home/<...>/.virtualenvs/other-sabt/local/lib/python2.7/site-packages/pip/__init__.py in <module>()
43 from pip.utils import get_installed_distributions, get_prog
44 from pip.utils import deprecation, dist_is_editable
---> 45 from pip.vcs import git, mercurial, subversion, bazaar # noqa
46 from pip.baseparser import ConfigOptionParser, UpdatingDefaultsHelpFormatter
47 from pip.commands import get_summaries, get_similar_commands
/home/<...>/.virtualenvs/other-sabt/local/lib/python2.7/site-packages/pip/vcs/mercurial.py in <module>()
7 from pip.utils import display_path, rmtree
8 from pip.vcs import vcs, VersionControl
----> 9 from pip.download import path_to_url
10 from pip._vendor.six.moves import configparser
11
/home/l<...>/.virtualenvs/other-sabt/local/lib/python2.7/site-packages/pip/download.py in <module>()
38 from pip.locations import write_delete_marker_file
39 from pip.vcs import vcs
---> 40 from pip._vendor import requests, six
41 from pip._vendor.requests.adapters import BaseAdapter, HTTPAdapter
42 from pip._vendor.requests.auth import AuthBase, HTTPBasicAuth
/home/<...>/.virtualenvs/other-sabt/local/lib/python2.7/site-packages/pip/_vendor/requests/__init__.py in <module>()
96
97 from . import utils
---> 98 from . import packages
99 from .models import Request, Response, PreparedRequest
100 from .api import request, get, head, post, patch, put, delete, options
/home/<...>/.virtualenvs/other-sabt/local/lib/python2.7/site-packages/pip/_vendor/requests/packages.py in <module>()
10 for mod in list(sys.modules):
11 if mod == package or mod.startswith(package + '.'):
---> 12 sys.modules['pip._vendor.requests.packages.' + mod] = sys.modules["pip._vendor." + mod]
13
14 # Kinda cool, though, right?
KeyError: 'pip._vendor.urllib3.contrib'
I'm seeing the same thing with pip 9.0.2 while running on the python:3.6 docker container
- pip version: 9.0.2
- python version: 3.6.4
- operating system: docker/linux
I've been able to narrow it down to occurring when the requests library is imported before the pip library. Here's an example:
docker-user:~# python
Python 3.6.4 (default, Mar 14 2018, 17:49:05)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> import pip
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.6/site-packages/pip/__init__.py", line 45, in <module>
from pip.vcs import git, mercurial, subversion, bazaar # noqa
File "/usr/local/lib/python3.6/site-packages/pip/vcs/mercurial.py", line 9, in <module>
from pip.download import path_to_url
File "/usr/local/lib/python3.6/site-packages/pip/download.py", line 40, in <module>
from pip._vendor import requests, six
File "/usr/local/lib/python3.6/site-packages/pip/_vendor/requests/__init__.py", line 98, in <module>
from . import packages
File "/usr/local/lib/python3.6/site-packages/pip/_vendor/requests/packages.py", line 12, in <module>
sys.modules['pip._vendor.requests.packages.' + mod] = sys.modules["pip._vendor." + mod]
KeyError: 'pip._vendor.urllib3.contrib'
>>> exit()
docker-user~# pip --version
pip 9.0.2 from /usr/local/lib/python3.6/site-packages (python 3.6)
The reverse works fine:
docker-user:~# python
Python 3.6.4 (default, Mar 14 2018, 17:49:05)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pip
>>> import requests
>>> print(pip)
<module 'pip' from '/usr/local/lib/python3.6/site-packages/pip/__init__.py'>
Are you importing pip in your code? Or is there a pip command you're running to trigger this?
In my case, I'm importing pip in my code to use pip.main().
Does it happen on the master branch as well?
(If you're using pip.main() that's been renamed to pip._internal.main() in master as we don't have any supported APIs).
Was just checking that. It looks like master works fine:
docker-user:~/libraries# python
Python 3.6.4 (default, Mar 14 2018, 17:49:05)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> import pip
>>> print(pip)
<module 'pip' from '/root/libraries/src/pip/__init__.py'>
>>> print(pip.__version__)
10.0.0.dev0
(good to know about the pip.main() refactor)
You'll need to import pip._internal, there's nothing in pip/__init__.py except the __version__.
Ah, yep. Seeing that now. Still seems to work:
docker-user:~/libraries# python
Python 3.6.4 (default, Mar 14 2018, 17:49:05)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> from pip import _internal
>>> print(_internal)
<module 'pip._internal' from '/root/libraries/src/pip/_internal/__init__.py'>
>>> import pip
>>> print(pip.__version__)
10.0.0.dev0
Alright. I'm inclined to say that that you'll want to either re-arrange your imports, or pin to 9.0.1. While I recognize that it sucks that it broke, we don't really support any usage of pip where you're importing pip itself (hence the refactor to move everything into pip._internal to make this crystal clear, since a lot of people were confused by this). In addition, we're planning on releasing in the upcoming weeks (see the announcement) which doesn't appear to have this problem.
This is largely because 9.0.x has drifted enough that cutting 9.0.2 was an ~8 hour process, and I'd much rather not have to do that again, particularly for an unsupported use of pip. If one of the other pip developers feel strongly about it, they're welcome to do it though.
Sorry again!
Yep, I can do that to solve my case. 👍
Understandable that it's not officially supported, but there's definitely usage of it out there. GitHub code search won't allow for using . so this search isn't perfect:
https://github.com/search?l=Python&q=import+pip+main%28&type=Code&utf8=%E2%9C%93
but it returns ~850k hits of import pip + main( on public github projects.
An anecdotal pattern I notice from a quick look through them, is that many are installing data packages. Not surprising since there are many data environments (notebook deployments especially) that obfuscate access to the host environment. This is serving as an in-python way of managing dependencies that can be shared in such contexts.
I'm not sure if there's any interest in maintaining the pip.main() as a passthrough command execution wrapper, but that could potentially mitigate the pain of upgrading.
Just a thought though, I've already refactored to resolve my issue.
I'm not sure if there's any interest in maintaining the
pip.main()as a passthrough command execution wrapper, but that could potentially mitigate the pain of upgrading.
I think this would make an excellent issue to discuss the relative merits of doing that (or not doing it).
FWIW I'm seeing something very similar just by calling pip install on some requirements:
++ pip install -U pip
Collecting pip
Using cached pip-9.0.2-py2.py3-none-any.whl
Installing collected packages: pip
Found existing installation: pip 9.0.1
Uninstalling pip-9.0.1:
Successfully uninstalled pip-9.0.1
Successfully installed pip-9.0.2
++ pip install -r requirements.pip
Collecting jellyfish==0.5.6 (from -r requirements.pip (line 1))
Using cached jellyfish-0.5.6.tar.gz
Collecting numpy==1.13.1 (from -r requirements.pip (line 2))
Using cached numpy-1.13.1-cp35-cp35m-manylinux1_x86_64.whl
Collecting vertica-python==0.7.3 (from -r requirements.pip (line 3))
Exception:
Traceback (most recent call last):
File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/commands/install.py", line 324, in run
requirement_set.prepare_files(finder)
File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/req/req_set.py", line 554, in _prepare_file
require_hashes
File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/req/req_install.py", line 278, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/index.py", line 465, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/index.py", line 423, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/index.py", line 568, in _get_pages
page = self._get_page(location)
File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/index.py", line 683, in _get_page
return HTMLPage.get_page(link, session=self.session)
File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/index.py", line 792, in get_page
"Cache-Control": "max-age=600",
File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/_vendor/requests/sessions.py", line 521, in get
return self.request('GET', url, **kwargs)
File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/download.py", line 386, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/_vendor/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/_vendor/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/_vendor/cachecontrol/adapter.py", line 37, in send
cached_response = self.controller.cached_request(request)
File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/_vendor/cachecontrol/controller.py", line 111, in cached_request
resp = self.serializer.loads(request, cache_data)
File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/_vendor/cachecontrol/serialize.py", line 114, in loads
return getattr(self, "_loads_v{0}".format(ver))(request, data)
File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/_vendor/cachecontrol/serialize.py", line 168, in _loads_v1
cached = pickle.loads(data)
ImportError: No module named 'pip._vendor.requests.packages.urllib3'; 'pip._vendor.requests.packages' is not a package
Strangely, I can avoid the error by calling pip install on my requirements with --no-cache-dir. So perhaps in my case there is some bad interaction involving pip 9.0.2 and cached packages?
Alright, well I was able to fix my problem by deleting the system pip cache. Now my jobs run as they were before, without needing to pin pip to 9.0.1 or specifying --no-cache-dir.
Dunno if this is related at all to what people are seeing here or on #5081, but noting my experience here nonetheless in case it helps someone.
👍 I experience the problem inside the build of docker containers based on python:2.7-slim, python:3.5-slim etc. So I cannot do finegrained control of --no-cache-dirs. 9.0.2 is already installed. I am calling pip.main() from inside a python script.
How to do it in windows??
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.