ImportError in system pip wrappers after an upgrade

Question

ImportError in system pip wrappers after an upgrade

pfmoore opened this issue 6 years ago · 122 comments

Overview

After upgrading to pip 10 or higher, many users are encountering error like

ImportError: cannot import name 'main'
TypeError: 'module' object is not callable
ImportError: No module named _internal
ModuleNotFoundError: No module named 'pip._internal'

These are caused by an incorrect attempt to upgrade pip, which has typically resulted in (parts of) multiple versions of pip being installed in the same Python installation, and those parts being incompatible.

It should be noted that these issues are invariably not problems with pip itself, but are due to incorrect use of pip, or unexpected interactions with system scripts that are not controlled by pip. So while we'll try to help you solve your issue, this is not the "fault" of pip, and you will have to be prepared to do at least some of the debugging and fixes on your own.

You can reach out to your Python provider (eg: Linux Distro, Cloud Provider, etc) for help with this issue.

General Advice

First, some general advice. It is assumed that anyone encountering issues will have failed to follow some part of this advice. Listing these items here is not intended to imply that "it's your fault and we won't help", but rather to help users to identify what went wrong, so that they can work out what to do next more easily.

Only ever use your system package manager to upgrade the system pip. The system installed pip is owned by the distribution, and if you don't use distribution-supplied tools to manage it, you will hit problems. Yes, we know pip says "you should upgrade with pip install -U pip" - that's true in a pip-managed installation, ideally distributions should patch this message to give appropriate instructions in the system pip, but they don't. We're working with them on this, but it's not going to happen soon (remember, we're looking at cases where people are upgrading old versions of pip here, so patches to new versions won't help).
Never use sudo with pip. This follows on from the first point. If you think you need to use sudo, you're probably trying to modify a distribution-owned file. See point 1.
Prefer to use --user. By doing this, you only ever install packages in your personal directories, and so you avoid interfering with the system copy of pip. But there are PATH issues you need to be aware of here. We'll cover these later. Put simply, it's possible to follow this advice, and still hit problems, because you're not actually running the wrapper you installed as --user.

Debugging the Issue

Before trying to work out what's going on, it's critically important that you understand precisely what scripts you are running and what versions of pip the relevant Python interpreter is using.

First, identify the full absolute path of the executable script that you are running. That's often in the traceback you get, but if it isn't, you can use OS tools like which, or Python's shutil.which to identify the right script. Watch out for your shell confusing the issue, with aliases or potentially stale hashed commands.

Once you have identified the script, make sure you can reproduce the problem using the absolute path to that script. If you can't, you probably found the wrong script, so check again.

Second, work out which version of Python the script is using to run pip. You'll often be able to get that from the shebang line of the script. This can often be the trickiest problem, as wrapper scripts can take many forms depending on what tool created them. In the worst case, you can simply make an intelligent guess at this point.

Once you know which Python is running pip, you can run python -m pip to invoke pip. In most cases, this will work fine, as it's the wrapper causing the issue, and not pip itself. Running pip via python -m in this way is often a perfectly acceptable workaround for the issue (at least in the short term).

Now run python -m pip --version. This will give you the exact version and location of the installation of pip that your Python is seeing.

At this point, you're usually done - the fundamental cause of all these problems is running a wrapper script which is written expecting to see a version of pip older than pip 10 (that's why it imports pip.main) under a Python interpreter that sees a copy of pip that's version 10 or later.

Fixing the Issue

The problem, of course, is fixing the issue. And that's where you really are on your own. If you have changed your system installation, you really need to put it back the way that the distribution installed it. That may well require an uninstall and reinstall of your distribution's pip package. Reinstalling is easy, of course - but uninstalling may require manually removing incorrect files. Or you may be able to force-reinstall with your distribution package manager, simply overwriting the incorrect files. Of course, this reverts you to the system-supplied version of pip. If you need a newer version, you should ask your distribution vendor, or use something like virtualenv to install it independently of your system packages.

It may be that you're simply running the "wrong" wrapper script. Maybe you did a --user install of a new version of pip, but your PATH is set to run the system version of the wrapper rather than the user-local one installed with pip. In that case, you can simply fix your PATH. That's usually the issue for people who do pip install --user --upgrade pip and get the pip.main error.

As already noted, python -m pip is a reliable workaround, at the cost of using a more verbose command to invoke pip.

Community Advice

The comments section of this issue is open for people to discuss specific problems, and to share potential solutions. Just to be clear, it's quite likely with problems of this nature that you'll need to modify system-supplied files or settings. You do so at your own risk. If you're not comfortable modifying your OS, or running as root, you should seek expert advice. To put it another way, if by following suggestions given here, you break your system, you get to keep the pieces. There's only so much that can be achieved remotely.

Also, the pip developers don't provide any guarantees that advice in the comments on this issue is correct, or that it won't damage your system. Again, use at your own risk.

Related Issues

The following issues have been reported which are related to this issue:
#5240, #5221, #5588, #5495, #5493, #5487, #5447, #5432, #5373, #5326, #5318, #5253

Answer 1 · 2018-08-05T11:10:12.000Z

The bug happens when not using virtualenv, using virtualenv fixes the problem on my side.
To fix the duplicate pip problem (described above) and downgrade to original pip version, I just rm -rf ~/.local/lib/python2.7/site-packages/ (beware!).

Answer 2 · 2018-08-13T19:26:10.000Z

I am pretty new to coding, however I managed to fix it in a kind of odd way.
My problem was that there was nothing in my /usr/bin/ folder regarding pip.
to fix this I had my buddy upload the get-pip.py program to raspberry pi, from which I downloaded the code to my computer and ran it. It then over-rode the old pip program and installed the new one.

Here's the code we used:
working computer to pi: scp /whatever_your_directory_is/get-pip.py pi@pi's_address:/pi's_directory
pi to my computer (the last period is important): scp pi@10.200.130.65:/pi's_directoryi/get-pip.py .
to download pip: sudo python get-pip.py

Sorry if it's weird, but that's what got it working for me

Answer 3 · 2018-08-17T17:49:52.000Z

After upgrading pip I ran into the error ImportError: cannot import name 'py31compat' when trying to import pyarrow. I have a conda installation and after rolling back pip to version 9 on all of my environments I still ran into the error.

The fix for me was to change line 72 in site-packages/pkg_resources/__init__.py from from . import py31compat to from tempfile import TemporaryDirectory.

setuptools/py31compat.py is a workaround for older versions of Python that don't have TemporaryDirectory. Since I'm running Python 3.6 this change eliminates the need to use py31compat.

Things seem to be working after this change but I'm not that confident it'll keep working down the road.

Answer 4 · 2018-08-17T22:25:00.000Z

The advice above:

  Only ever use your system package manager to upgrade pip.

conflicts with the cheerful siren call:

You are using pip version 8.1.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

This seems to be some kind of tarpit :-/

Answer 5 · 2018-08-17T23:18:30.000Z

@ErichBSchulz We're aware of that - see #5346.

Maybe I should have more explicitly said "Only ever use your system package manager to upgrade your system pip". The more detailed comment in that bullet point makes it clearer, but I've edited the post to make it explicit.

Answer 6 · 2018-08-19T21:40:08.000Z

I followed some instructions elsewhere and ran

sudo apt install python-pip
pip install --upgrade pip

and hit upon this issue.

python -m pip uninstall pip

fixed it, returning me to the system pip

I have to say, although I don't fully understand the complications involved, it doesn't feel like pip should self-immolate like this.

Answer 7 · 2018-08-27T18:15:24.000Z

I would add this advice of not upgrading your system pip in the documentation.
I just followed these instructions and killed my raspbian pip and pip3 installs.

On the page:
https://packaging.python.org/tutorials/installing-packages/#ensure-pip-setuptools-and-wheel-are-up-to-date it lists python -m pip install --upgrade pip setuptools wheel , which is in conflict with recommentation 1 and 3 in this issue.

I think I restored that with:

python3 -m pip uninstall pip setuptools wheel
sudo apt-get --reinstall install  python3-setuptools python3-wheel python3-pip

python -m pip uninstall pip setuptools wheel
sudo apt-get --reinstall install python-setuptools python-wheel python-pip

After that my raspbian PI zero pip and pip3 report the ancient 9.0.1 versions, which is the latests available according to https://tracker.debian.org/pkg/python-pip
The request to update pip in debian repo can be found here:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=901393

Answer 8 · 2018-09-10T07:57:29.000Z

pip3 install --user pip seems to break my system pip.

Isn't it a good idea to install the latest version of pip in the user's home directory?

Answer 9 · 2018-09-12T05:27:47.000Z

The Ubuntu's python-pip-9.0.1 installed(via apt get python-pip) is conflicting with Latest pip-18.0 provided by pypa. So sudo rm -rf ~/.local/lib/python2.7/site-packages/(Beware!) removes the ubuntu's python-pip and resolves the problem. Only one pip can exist on the system. Thanks @denisglotov

Also Upgrading pip doesn't help as it does'nt overwrite the preexisting installed python-pip but instead create another copy of pip in different directory causing the conflict.

Answer 10 · 2018-09-13T21:44:20.000Z

My problem was installing pip in ~/.local/ and having at the end of $PATH.

Answer 11 · 2018-09-21T20:15:19.000Z

I wonder how npm and cpan solve this problem? In any case, I just uninstalled system pip3 and created my own:

#!/bin/sh

exec python3 -m pip "$@"

I'm the only user running it on this system anyway. Victory!

Answer 12 · 2018-09-29T17:49:25.000Z

Related discussion (from April) with a variety of solutions from Stackoverflow community:
https://stackoverflow.com/questions/49836676/error-after-upgrading-pip-cannot-import-name-main
https://stackoverflow.com/questions/49964093
Let's hope #5346 will help minimize future impacts, as OS distributions alone may be too slow at preventing this issue from happening in their LTS and later releases..

Answer 13 · 2018-10-18T09:07:18.000Z

...

Debugging the Issue

...
Second, work out which version of Python the script is using to run pip. You'll often be able to get that from the shebang line of the script. This can often be the trickiest problem, as wrapper scripts can take many forms depending on what tool created them. In the worst case, you can simply make an intelligent guess at this point.

Once you know which Python is running pip, you can run python -m pip to invoke pip. In most cases, this will work fine, as it's the wrapper causing the issue, and not pip itself. Running pip via python -m in this way is often a perfectly acceptable workaround for the issue (at least in the short term).
...

Well:

$ python -m pip
/usr/bin/python: No module named pip
$ python3 -m pip3
/usr/bin/python3: No module named pip3

OK, so let's try something else:

$ python3 -m pip uninstall pip3
Skipping pip3 as it is not installed.
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Wait,
'You are using pip version 10.0.1, however version 18.1 is available.'

What?

$ dpkg -l '*pip*'
...                                                                  Version                                   Architecture                              Description
+++-=========================================================================-=========================================-=========================================-=======================================================================================================================================================
ii  libpipeline1:amd64                                                        1.4.1-2                                   amd64                                     pipeline manipulation library
un  pppdcapiplugin                                                            <none>                                    <none>                                    (no description available)
$

So what am I supposed to do now?

PS I really consider the sentence:
'You are using pip version 10.0.1, however version 18.1 is available.' really, really confusing...

Answer 14 · 2018-10-18T09:22:22.000Z

...

Update:
Finally I've managed to fix my system using the #5599 (comment).
Thank you very much!

Xubuntu

Distributor ID: Ubuntu
Description:    Ubuntu 16.04.5 LTS
Release:        16.04
Codename:       xenial

Answer 15 · 2018-11-21T18:17:11.000Z

I just ran into this problem. I believe my problem was because of the referenced use of --user since the python shebang path in the pip wrapper was set to: #!/usr/local/opt/python3/bin/python3.6, yet all my python packages and python installs are in /usr/local/bin with sym links to homebrew directories /usr/local/Cellar/.... I was going to edit the pip paths, but decided to preference an uninstall & new installation of pip with these steps:

remove pip: sudo pip uninstall pip
removed leftover pip packages from /usr/local/bin/ and /usr/local/Cellar/python3
installed updates to excode xcode-select --install
updated brew brew update
upgraded python3 to 3.7.3 brew upgrade python

I was going to uninstall python3 with brew, but since I needed to also update my python from 3.6.3 to 3.7.1, I just skipped the removal of older python3 versions.

Answer 16 · 2018-12-14T21:09:57.000Z

I was installing devstack (on CentOS on VirtualBox) when I hit this error.
I have a root user and a devstack user. devstack user pip version was 18.1
[stack@DevStack devstack]$ python -m pip --version
pip 18.1 from /opt/stack/.local/lib/python2.7/site-packages/pip (python 2.7)
but stack install was downgrading root users pip version to 9.x (really annoying!)
[root@DevStack ~]# python -m pip --version
pip 9.0.3 from /usr/lib/python2.7/site-packages (python 2.7)

I noticed in the devstack install:
Collecting pip!=8,<10 (from -c /opt/stack/repos/devstack/tools/cap-pip.txt (line 1))
So changed cap-pip.txt to
pip!=17,<19

Not sure why devstack are enforcing a version

Answer 17 · 2019-01-03T09:03:10.000Z

I ran into the same problem.
It seems that there is a conflict between pip in system and pip in python site-packages for me in my ubuntu.
After upgrading the pip as the pip command suggests, I removed the system pip using
sudo apt remove python-pip
And restarting the bash, the error goes away.

Answer 18 · 2019-01-04T18:55:07.000Z

@pfmoore

In this comment you said "using python -m pip it's still not supported". The documentation instead says to call pip from another script like this:

subprocess.check_call([sys.executable, '-m', 'pip', 'install', 'my_package'])

which of course will ultimately call python -m pip. So who is right?

You also said that "from pip import main was never supported". So what is the correct way to call pip? The documentation says to call

$ pip <pip arguments>

but that of course will call the unsupported distribution-specific wrapper, which, in the case of Ubuntu, uses the unsupported .from pip import main

Answer 19 · 2019-01-04T19:11:13.000Z

@pfmoore
3. I have installed pip in a virtualenv using the get-pip.py script, a method described in the documentation, but in a virtual environment as not to conflict with the distribution's pip:

virtualenv -p python3 --no-pip pipinstall
source ./pipinstall/bin/activate
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python3 get-pip.py
cat ./pipinstall/bin/pip3.6

Here is the content of the file ./pipinstall/bin/pip3.6

#!/home/raffaele/bin/pyvenv/pipinstall/bin/python3

# -*- coding: utf-8 -*-
import re
import sys

from pip._internal import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

The unsupported from pip._internal import main is being used instead of subprocess.call().

Whose bug is it?

Answer 20 · 2019-01-04T20:01:40.000Z

pip3.6 is a script installed by (and part of) pip itself. So it is allowed to use pip internals.

Ubuntu supply their own pip wrapper script, which means they take on the support burden of using (and supporting/maintaining) calls to undocumented internal functions.

Answer 21 · 2019-01-05T17:14:59.000Z

I ran into the same problem.
It seems that there is a conflict between pip in system and pip in python site-packages for me in my ubuntu.
After upgrading the pip as the pip command suggests, I removed the system pip using
sudo apt remove python-pip
And restarting the bash, the error goes away.

This solution worked for me. Restarting bash was what I was missing before. Messy!

Answer 22 · 2019-01-13T13:13:44.000Z

I have a more common request but related to that issue.
I tried uprgrading pip in the not adviced way and run into that main error problem.

I will repair my system python according to the advices (thx very much for the detail description, helped a lot) but now I have a question related to this. Sorry in advance, I would not consider myself very experienced, so I dont know if this is the wrong issue to post this or if the question is redundant. But still it is a question to circumvent the problem caused by upgrading the system pip.

So my question is: is it possible to install a python version next to the system python and use this as default under the following conditions:

i dont want the alternative python version to interfer in any way with the system python
- this should assure that I dont have any problems with pip and I can really upgrade the non system pip to version 18
not using any virtual environments
- I dont want to do a source /.../activate all the time i want to use my dedicated python default version

Related to that:

if i install a python version using a tar downloaded from python.org and install it via make altinstall to make it an alternative installation
- is it without any risk possible to change the default system python paths to the ones of the alternative installation or redefine the aliases to use the alternative installation?
- if I then use pip of the alternative installation is it really installing only to the alternative installation or does it interfer with the system python?
- is it even possible if I have python 2.7 and 3.6 as my system python to install them again as an alternative python version and use them as default?

I tried to search the internet for these questions and found mostly things related to envs and mutliple python installations with different versions but not the same. I also stumbled upon pipenv but this seems again to be just the same as using envs.

Regarding the --user statement, if I got it right can be used to install a newer version of pip for the current user. Then I have to adjust the paths of the default pip to that one, right? And does it require already a sort of alternative python version installed for the current user?

Something like installing an alternative python for a user is probably what I am looking for and make the user`s python the default python for that user. But I did not find issues or topics concretely addressing this.

If someone has an idea about this and can share it with me I would be very thankful. And maybe this helps also others to set up a default python that can be altered in any desired way without interfering with the system python.

Answer 23 · 2019-01-14T07:29:42.000Z

I ran into this same issue and was able to solve with this advice:

#5599 (comment)

Thanks much @denisglotov and @abhiTronix

Answer 24 · 2019-02-05T09:10:43.000Z

One thing that I've run across is that I'm using 3.7.2 which currently isn't included with Ubuntu's default package manager.
Also, every time I have tried to upgrade pip using non-sudo, I've always received an error. Since 3.7 isn't officially available (that I'm currently aware of), I built both 3.7.2 and pip3 from source and then ensured that files in my /usr/local/lib/python3.7 directory were all owned by my user and not root.

I guess that's why it's also recommended to use a virtualenv as to not mess up your systems default version?

Answer 25 · 2019-02-09T03:28:55.000Z

I fixed this by opening the files /usr/local/bin/pip and /usr/bin/pip and replacing the line from pip import main with from pip._internal import main.

Answer 26 · 2019-02-18T23:54:49.000Z

Did you try

$ hash -d pip

Or in dash (sh):

$ hash -r pip

Answer 27 · 2019-02-21T14:45:31.000Z

Summary: How can i rewrite pip using subprocess?

I'm trying to install the R package keras from RStudio. Under the hood, it uses Python. When I run the R function install_keras(), it hits the infamous ImportError: cannot import name main.

@pfmoore :

The absolute path of the executable script is ~/.virtualenvs/r-tensorflow/bin/pip
I can reproduce the problem using the absolute path to that script.
The script is using ~/.virtualenvs/r-tensorflow/bin/python, and -V reports Python 2.7.6
Running python -m pip emits a lengthy Usage page; apparently <command> is missing.
Running python -m pip --version reports pip 19.0.3 from /home/brech/.virtualenvs/r-tensorflow/local/lib/python2.7/site-packages/pip (python 2.7), where /home/brech is ~.

So it's all local to my r-tensorflow, installed this week. No upgrade or system pip seems involved.

You give two pieces of advice: (A) clear the hash, or (B) rewrite using subprocess.

In a shell, I do

$ hash -d pip
bash: hash: pip: not found

So how do I rewrite ~/.virtualenvs/r-tensorflow/bin/pip using subprocess instead of importing from pip? Here is the complete script:

#!/home/brech/.virtualenvs/r-tensorflow/bin/python

# -*- coding: utf-8 -*-
import re
import sys

from pip import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

Answer 28 · 2019-02-21T14:54:25.000Z

@Quiigi That pip script is not the one shipped with pip. You should discuss the issue with whoever or whatever generated that script.

To give you some suggestions:

As it's in a .virtualenvs directory, presumably pip (and that script) was installed when the virtualenv was created. Start looking there - how did the virtualenv get created?
The script looks like the sort of thing easy_install creates - did you (or whatever built the environment) install pip using easy_install? We don't support that, so you should install pip "properly" (via virtualenv, or via get-pip.py)

This actually looks suspiciously like something I've seen before, with PyCharm. Are you using PyCharm? If so there's a known bug in their "build an environment" stuff that installs pip using easy_install, so that may be your issue here. I raised a bug with them about it, but I don't know the bug ID (I stopped caring about PyCharm because they didn't seem interested in fixing this issue :-))

Answer 29 · 2019-02-26T15:01:12.000Z

Traceback (most recent call last):
File "/usr/bin/pip", line 9, in
from pip import main
ImportError: cannot import name main..

I am using virtulaenv and there is only one pip , and still getting this error while useing pip.

Answer 30 · 2019-02-26T18:49:36.000Z

@pfmoore It's from tensorflow, not PyCharm. See install.R; below are the two functions that look most relevant to me. And tensorflow shoots itself in the foot where it calls pip_install three times:

This code finds that the native pip 1.5.4 is older than 8.1, so it tries to upgrade 3 packages. First pip, which works because the pip script that tensorflow installed into my virtualenv, and which I reproduced above, is compatible with 1.5.4, and thus upgrades to pip 19.0.3.

Next (still because pip 1.5 is older than 8.1) it tries to upgrade wheel, but now the same supplied script ~/.virtualenvs/r-tensorflow/bin/pip is incompatible with the just-installed pip 19.0.3 from ~/.virtualenvs/r-tensorflow/local/lib/python2.7/site-packages/pip, which no longer provides main. So the second invocation of pip fails, and thus the whole installation of tensorflow!

  # function to call pip within virtual env
  pip_install <- function(pkgs, message) {
    cmd <- sprintf("%ssource %s && %s install --ignore-installed --upgrade %s%s",
                   ifelse(is_osx(), "", "/bin/bash -c \""),
                   shQuote(path.expand(virtualenv_bin("activate"))),
                   shQuote(path.expand(virtualenv_bin(pip_version))),
                   paste(shQuote(pkgs), collapse = " "),
                   ifelse(is_osx(), "", "\""))
    cat(message, "...\n")
    result <- system(cmd)
    if (result != 0L)
      stop("Error ", result, " occurred installing TensorFlow", call. = FALSE)
  }

  # upgrade pip and related utilities if its older than 8.1
  if (installed_pip_version() < "8.1") {
    pip_install("pip", "Upgrading pip")
    pip_install("wheel", "Upgrading wheel")
    pip_install("setuptools", "Upgrading setuptools")
  }

My workaround was to do the following in a shell:

   $ ~/.virtualenvs/r-tensorflow/bin/python -m pip install \
      --ignore-installed --upgrade tensorflow==1.12.0 h5py pyyaml requests \
      Pillow keras tensorflow-hub tensorflow-probability scipy

Answer 31 · 2019-02-26T19:17:51.000Z

@pfmoore It's from tensorflow

OK, so you should probably raise this with them. It looks like their install process is writing an incorrect form of the pip wrapper script.

If that script is upgrading the system-supplied pip using pip (rather than using the standard OS packages) then that's not supported, so again that's a problem with the tensorflow supplied install script (which, if it's modifying the system pip, you're presumably running with sudo, which is also not recommended/supported).

Anyhow, the main point is that this appears to be a tensorflow issue, not a pip issue.

Answer 32 · 2019-03-21T14:43:27.000Z

ln -sf /usr/local/bin/pip3 /usr/bin/pip works.

Answer 33 · 2019-06-13T19:08:42.000Z

So, I had this problem, and apparently it was because i have a local copy installed along with the system version - removing my local version: rm ~/.local/bin/pip3* and then reinstalling from repo sudo apt install --reinstall python3-pip worked for me.

Answer 34 · 2019-07-16T17:03:40.000Z

(For ubuntu users) Final solution! No action required!
You may want to upgrade pip using
python3 -m pip install --user --upgrade pip
But error occurs!
This because new pip locates in $HOME/.local/bin but your $PATH doesn't point it.
But actually .profile have a script to load /.local/bin to path!
So just logout and login again.
Then problem fixed.
By the way, it seems make sense to have pip packages in user level and load it via .profile because this also for user level.

Answer 35 · 2019-10-07T20:31:00.000Z

It seems to me that the advice: Only ever use your system package manager to upgrade the system pip. isn't really sufficient: commands like pip3 install pipenv will cause pip to be upgraded, stopping it (or rather, the ubuntu 16.04 wrapper) from working.

Answer 36 · 2019-10-14T22:35:02.000Z

In 19.3 we moved the main function from pip._internal to pip._internal.main, so you may see output like:

Traceback (most recent call last):
  File "/usr/bin/pip3", line 16, in <module>
    sys.exit(main())
TypeError: 'module' object is not callable

The advice in the original post still holds.

Answer 37 · 2019-10-15T05:55:51.000Z

That new output is now occurring when prior versions succeeded, breaking CI at https://github.com/theacodes/nox/

It looks like

$ pip install wheel
Traceback (most recent call last):
  File "c:\python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Python37\Scripts\pip.exe\__main__.py", line 9, in <module>
TypeError: 'module' object is not callable

See https://ci.appveyor.com/project/theacodes/nox/builds/28111820/job/64e3ug9ibm0vkteo

Answer 38 · 2019-10-15T12:08:26.000Z

Sharing, if anyone else coming across this as of 19.3:

$ python -m pip --version
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/circleci/venv/lib/python3.6/site-packages/pip/__main__.py", line 16, in <module>
    from pip._internal import main as _main  # isort:skip # noqa
  File "/home/circleci/venv/lib/python3.6/site-packages/pip/_internal/__init__.py", line 40, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/home/circleci/venv/lib/python3.6/site-packages/pip/_internal/cli/autocompletion.py", line 8, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/home/circleci/venv/lib/python3.6/site-packages/pip/_internal/cli/main_parser.py", line 11, in <module>
    from pip._internal.commands import (
  File "/home/circleci/venv/lib/python3.6/site-packages/pip/_internal/commands/__init__.py", line 9, in <module>
    from pip._internal.commands.download import DownloadCommand
  File "/home/circleci/venv/lib/python3.6/site-packages/pip/_internal/commands/download.py", line 10, in <module>
    from pip._internal.operations.prepare import RequirementPreparer
  File "/home/circleci/venv/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 9, in <module>
    from pip._internal.distributions import (
  File "/home/circleci/venv/lib/python3.6/site-packages/pip/_internal/distributions/__init__.py", line 1, in <module>
    from pip._internal.distributions.source import SourceDistribution
ImportError: cannot import name 'SourceDistribution'

Answer 39 · 2019-10-15T12:35:40.000Z

@pfmoore It might not be the best place to ask, but I'll try anyway.

I suspect this repeating issue, of maybe a badly configured /workflow for pip installs is a repeating theme and more specifically highlighted on CI systems.

For the requirements of:

Start a new test (for TravisCI, CircleCI and friends means run a new docker instance)
Install a python version (or use the existing one from the image)
Create a virtual environment.
Upgrade pip to latest (or to explicit version).
Install the specific dependencies of the software being tested.

Maybe there should be a cook-book documented on how to achieve that?

Answer 40 · 2019-10-15T12:42:34.000Z

@AvnerCohen That sounds like a very good idea (specifically "setting up pip on common CI systems"). It's probably something that could be written as a subsection within the "User Guide" section of the pip documentation. Would you be interested in contributing such a section?

Going beyond CI systems to cover all of the various possibilities for user environments in general is too much, though. We have sections on installing and using pip, and I think it's reasonable to write them based on the assumption that the user knows how to use and manage their computing environment. (That's not actually true in many cases, people often don't understand their environments for perfectly acceptable reasons, but I don't think it's the place of the pip manual to cover that general issue).

Answer 41 · 2019-10-15T13:11:24.000Z

Thanks @pfmoore .
Would love to help on that, but I think what I am doing right now is wrong since we keep coming across such issues. so not sure this should be presented as best practice.

I guess it would be best to discuss this over a dedicated PR, I'll stub something, taking into account also the "General Advice" and "Fixing the Issue" from this ticket and hopefully we can create something useful for others as well.

Answer 42 · 2019-10-15T13:56:41.000Z

OK, cool. I think one of the problems here is that no-one knows what "best practice" is (and everyone's quite reasonably focused on fixing their own problem, not working out a general solution), which is why we keep getting the same sort of questions. So having somewhere we can thrash out the "right" solution will be a good step forward.

Answer 43 · 2019-10-26T15:24:15.000Z

I followed some instructions elsewhere and ran
sudo apt install python-pip
pip install --upgrade pip
and hit upon this issue.
python -m pip uninstall pip
fixed it, returning me to the system pip

I have to say, although I don't fully understand the complications involved, it doesn't feel like pip should self-immolate like this.

You will need to clear the cache since the terminal will give : pip: No such file or directory

$ hash -r

Answer 44 · 2019-11-14T15:02:10.000Z

@pradyunsg Any plans to fix this? This issue is open since more than a year ago, an certainly quite critical for pip3 and all linux users!

If for whatever reason it cannot be fixed, an "official" statement on how we are supposed to upgrade would be great.

Answer 45 · 2019-11-14T15:20:51.000Z

@Oteph There's nothing to fix here. See the original issue description, specifically:

It should be noted that these issues are invariably not problems with pip itself, but are due to incorrect use of pip, or unexpected interactions with system scripts that are not controlled by pip.

We can't provide an "official" statement on how to upgrade scripts (system pip wrappers) that we didn't provide and that we don't maintain.

Answer 46 · 2019-11-14T16:02:45.000Z

@pfmoore
Thanks for the clarification!

Here is how i fixed it for myself:

Assuming you installed the newest version of pip with
pip3 install --upgrade pip --user
and it is now broken with the dreadful ImportError: cannot import name 'main'

This is how you can get a working installation:

echo "export PATH=~/.local/bin:$PATH" >> ~/.bashrc
source ~/.bashrc

Now pip3 should now work as expected. Note that this only masks the issue.

If it is not yet broken execute:

pip3 install --upgrade pip --user 
echo "export PATH=~/.local/bin:$PATH" >> ~/.bashrc
source ~/.bashrc

Answer 47 · 2019-12-28T10:07:10.000Z

So to use the latest version of pip, we have to install a local (per-user) version of pip, and leave the system-installed pip untouched. This is clear and helpful advice. Thanks. I am very lucky I saw this page in time, before using sudo to install.

But ... what a mess. On a system with N users, there could be N separately-installed versions of pip, some of them identical. Spasmodically, all N users will decide they should have the latest and greatest version of pip, and upgrade their copies asynchronously. Creating this situation was a massive blunder. I'm sure it's extremely difficult to fix now, but it's really important!

Answer 48 · 2019-12-28T11:02:15.000Z

There is a simple way to “solve” this: Upgrade the system pip with the system package manager. If you can (say) apt upgrade to the latest pip, everyone can use that. If you can’t, the problem is in the system, and per-user installs is the best pip maintainers can offer.

Answer 49 · 2019-12-30T08:04:44.000Z

I had the same experience with this issue #5373.
The only solution for me was uninstalling the pip.

sudo apt remove python3-pip
sudo apt purge python3-pip

And follow the installation guide on this site https://pip.pypa.io/en/stable/installing/
Now pip works like a charm.

Answer 50 · 2020-01-08T08:34:37.000Z

I had some related problem in Ubuntu in WSL. I had installed pip with --user and since --upgrade, modules were not found. There weren't many system-level pip packages installed (just docker-compose), so I decided to fix my system pip from scratch using apt-get.

apt-get installs a much older version of pip which can lead to
problems, completely remove python-pip with:
apt-get remove --purge python-pip
then:
curl https://bootstrap.pypa.io/get-pip.py | sudo python
using sudo if required

Answer 51 · 2020-01-21T12:50:45.000Z

Please, have travis piepline or something that covers that kind of situation, and not release it before.

Answer 52 · 2020-01-21T17:17:00.000Z

#7499 has been done, to partially mitigate this problem for now. We will remove those additional wrappers at a later date, so please treat the warning that pip spews out about "old wrappers" as serious and solve the underlying issues (or at least help by identifying them and letting us know here).

Answer 53 · 2020-01-25T18:59:26.000Z

To anyone who finds this issue and beats their head against trying to upgrade the system pip to something past their distro's version -- just don't bother. There doesn't seem to be any clear path to upgrade the entire thing.

What did seem to work was creating a virtual environment using virtualenv for my project and using the pip there.

Instructions are very simple:

virtualenv --python=python3 my-project-name
source my-project-name/bin/activate

Now pip --version reports the latest, and pip install -U pip works just as expected. The only downside is that packages installed by this pip will only be available when using the virtual environment.

To stop using the virtual environment, just type deactivate.

I know this isn't a solution for everyone, but this is overall cleaner if you're like me and just trying to use pip to install one damn package for your project.

Answer 54 · 2020-01-29T08:32:11.000Z

I would add this advice of not upgrading your system pip in the documentation.
I just followed these instructions and killed my raspbian pip and pip3 installs.

On the page:
https://packaging.python.org/tutorials/installing-packages/#ensure-pip-setuptools-and-wheel-are-up-to-date it lists python -m pip install --upgrade pip setuptools wheel , which is in conflict with recommentation 1 and 3 in this issue.

I think I restored that with:
python3 -m pip uninstall pip setuptools wheel
sudo apt-get --reinstall install  python3-setuptools python3-wheel python3-pip

python -m pip uninstall pip setuptools wheel
sudo apt-get --reinstall install python-setuptools python-wheel python-pip 
After that my raspbian PI zero pip and pip3 report the ancient 9.0.1 versions, which is the latests available according to https://tracker.debian.org/pkg/python-pip
The request to update pip in debian repo can be found here:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=901393

Thanks for solving the issue.

Answer 55 · 2020-02-28T11:27:22.000Z

To fix the PATH run Command Prompt as administrator:
pip install --upgrade --force-reinstall --ignore-installed pip
that fixed the issue

Answer 56 · 2020-02-28T13:00:37.000Z

@millerbird0 Can you please add info why this fixes the issue? My question is --ignore-installed will ignore if a package is installed right? Then, --force-reinstall shall uninstall it first if already installed?

Answer 57 · 2020-02-28T13:20:21.000Z

@millerbird0 Can you please add info why this fixes the issue? My question is --ignore-installed will ignore if a package is installed right? Then, --force-reinstall shall uninstall it first if already installed?

@ZoranPandovski it worked for me. When you run the script as administrator it will fix the PATH
I updated the pip as a local user using --user install that is where problem started, so i how to reinstall it as administrator to fix the problem with PATH. --force-reinstall --ignore-installed otherwise it will give you an error when you try to install.

Answer 58 · 2020-03-10T10:27:41.000Z

Type
python -m pip install anything
then it's working

Answer 59 · 2020-03-10T10:38:10.000Z

TLDR: Does this all mean that from now on we will always have to use python -m pip install ... instead of pip install ... ?

Answer 60 · 2020-03-10T11:42:23.000Z

@RafalSkolasinski It's far safer, because you know you're getting the version of pip installed with the version of Python that you're running. Using the executable wrappers involves

Taking care of any PATH issues to ensure that the wrapper you expect is getting run.
The possibility that a vendor-modified pip executable is being used rather than the one generated by the version of pip you want to run.

Many errors that we've seen boil down to "you aren't actually running what you think you are running" - either because people are inadvertently running the wrong wrapper, or because they are running a wrapper that isn't created (and hence isn't supported) by pip. Using python -m pip avoids these.

Look at it this way - if python -m pip fixes the issue, then the problem isn't with pip. So fixing the wrappers is going to need either changes to your environment (PATH, most likely) or fixes from your distro vendor. Whether you want to work on a "proper" fix, given that python -m pip is a workaround, is up to you.

Personally, I think people would find it far easier to just use python -m pip. But changing years of muscle memory isn't easy, so I doubt that will happen quickly...

Answer 61 · 2020-03-10T12:01:52.000Z

Thanks @pfmoore for a brief and nice summary. I think it makes a lot of sense.

Answer 62 · 2020-03-27T12:22:06.000Z

Overview

After upgrading to pip 10 or higher, many users are encountering error like
* `ImportError: cannot import name 'main'`

* [`TypeError: 'module' object is not callable`](https://github.com/pypa/pip/issues/5599#issuecomment-541960822)

* [`ImportError: No module named _internal`](https://github.com/pypa/pip/issues/5253#issue-314720523)

* [`ModuleNotFoundError: No module named 'pip._internal'`](https://github.com/pypa/pip/issues/5373#issue-320586093)
These are caused by an incorrect attempt to upgrade pip, which has typically resulted in (parts of) multiple versions of pip being installed in the same Python installation, and those parts being incompatible.

It should be noted that these issues are invariably not problems with pip itself, but are due to incorrect use of pip, or unexpected interactions with system scripts that are not controlled by pip. So while we'll try to help you solve your issue, this is not the "fault" of pip, and you will have to be prepared to do at least some of the debugging and fixes on your own.

General Advice

First, some general advice. It is assumed that anyone encountering issues will have failed to follow some part of this advice. Listing these items here is not intended to imply that "it's your fault and we won't help", but rather to help users to identify what went wrong, so that they can work out what to do next more easily.
1. **Only ever use your system package manager to upgrade the system pip**. The system installed pip is owned by the distribution, and if you don't use distribution-supplied tools to manage it, you will hit problems. Yes, we know pip says "you should upgrade with pip install -U pip" - that's true in a pip-managed installation, ideally distributions should patch this message to give appropriate instructions in the system pip, but they don't. We're working with them on this, but it's not going to happen soon (remember, we're looking at cases where people are upgrading old versions of pip here, so patches to new versions won't help).

2. **Never use sudo with pip**. This follows on from the first point. If you think you need to use sudo, you're probably trying to modify a distribution-owned file. See point 1.

3. **Prefer to use `--user`**. By doing this, you only ever install packages in your personal directories, and so you avoid interfering with the system copy of pip. But there are `PATH` issues you need to be aware of here. We'll cover these later. Put simply, it's possible to follow this advice, and still hit problems, because you're not actually running the wrapper you installed as `--user`.
Debugging the Issue

Before trying to work out what's going on, it's critically important that you understand precisely what scripts you are running and what versions of pip the relevant Python interpreter is using.

First, identify the full absolute path of the executable script that you are running. That's often in the traceback you get, but if it isn't, you can use OS tools like which, or Python's shutil.which to identify the right script. Watch out for your shell confusing the issue, with aliases or potentially stale hashed commands.

Once you have identified the script, make sure you can reproduce the problem using the absolute path to that script. If you can't, you probably found the wrong script, so check again.

Second, work out which version of Python the script is using to run pip. You'll often be able to get that from the shebang line of the script. This can often be the trickiest problem, as wrapper scripts can take many forms depending on what tool created them. In the worst case, you can simply make an intelligent guess at this point.

Once you know which Python is running pip, you can run python -m pip to invoke pip. In most cases, this will work fine, as it's the wrapper causing the issue, and not pip itself. Running pip via python -m in this way is often a perfectly acceptable workaround for the issue (at least in the short term).

Now run python -m pip --version. This will give you the exact version and location of the installation of pip that your Python is seeing.

At this point, you're usually done - the fundamental cause of all these problems is running a wrapper script which is written expecting to see a version of pip older than pip 10 (that's why it imports pip.main) under a Python interpreter that sees a copy of pip that's version 10 or later.

Fixing the Issue

The problem, of course, is fixing the issue. And that's where you really are on your own. If you have changed your system installation, you really need to put it back the way that the distribution installed it. That may well require an uninstall and reinstall of your distribution's pip package. Reinstalling is easy, of course - but uninstalling may require manually removing incorrect files. Or you may be able to force-reinstall with your distribution package manager, simply overwriting the incorrect files. Of course, this reverts you to the system-supplied version of pip. If you need a newer version, you should ask your distribution vendor, or use something like virtualenv to install it independently of your system packages.

It may be that you're simply running the "wrong" wrapper script. Maybe you did a --user install of a new version of pip, but your PATH is set to run the system version of the wrapper rather than the user-local one installed with pip. In that case, you can simply fix your PATH. That's usually the issue for people who do pip install --user --upgrade pip and get the pip.main error.

As already noted, python -m pip is a reliable workaround, at the cost of using a more verbose command to invoke pip.

Community Advice

The comments section of this issue is open for people to discuss specific problems, and to share potential solutions. Just to be clear, it's quite likely with problems of this nature that you'll need to modify system-supplied files or settings. You do so at your own risk. If you're not comfortable modifying your OS, or running as root, you should seek expert advice. To put it another way, if by following suggestions given here, you break your system, you get to keep the pieces. There's only so much that can be achieved remotely.

Also, the pip developers don't provide any guarantees that advice in the comments on this issue is correct, or that it won't damage your system. Again, use at your own risk.

Related Issues

The following issues have been reported which are related to this issue:
#5240, #5221, #5588, #5495, #5493, #5487, #5447, #5432, #5373, #5326, #5318, #5253

python -m pip this takes me to pip help!, Does pip invoking is known this ?

Answer 63 · 2020-03-27T12:25:26.000Z

amshashankk commented 5 years ago

Answer 64 · 2020-03-27T15:58:37.000Z

WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see #5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
Requirement already satisfied: youtube-dl in c:\users\dell\appdata\local\programs\python\python38-32\lib\site-packages (2020.3.24)

Answer 65 · 2020-03-29T17:23:58.000Z

Just delete the old script wrapper in PATH after upgrade on my macos.

$ pip3 --version
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
pip 20.0.2 from /Library/Python/3.7/site-packages/pip (python 3.7)

$ which pip3
/usr/bin/pip3

$ /usr/local/bin/pip3 --version
pip 20.0.2 from /Library/Python/3.7/site-packages/pip (python 3.7)

$ sudo mv /usr/bin/pip3 /usr/bin/old_pip3
$ pip3 --version
pip 20.0.2 from /Library/Python/3.7/site-packages/pip (python 3.7)

Answer 66 · 2020-03-29T17:37:46.000Z

Thank you very much for that :)

…

On Sun, 29 Mar 2020, 10:54 pm chenbc, ***@***.***> wrote: Just delete the old script wrapper in PATH after upgrade on my macos. $ pip3 --version WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip. Please see #5599 for advice on fixing the underlying issue. To avoid this problem you can invoke Python with '-m pip' instead of running pip directly. pip 20.0.2 from /Library/Python/3.7/site-packages/pip (python 3.7) $ which pip3 /usr/bin/pip3 $ /usr/local/bin/pip3 --version pip 20.0.2 from /Library/Python/3.7/site-packages/pip (python 3.7) $ sudo mv /usr/bin/pip3 /usr/bin/old_pip3 $ pip3 --version pip 20.0.2 from /Library/Python/3.7/site-packages/pip (python 3.7) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5599 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO62MGAJIOBLD3EO4AHOB3DRJ5737ANCNFSM4FJ4EQWA> .

Answer 67 · 2020-04-03T21:44:49.000Z

To fix the PATH run Command Prompt as administrator:
pip install --upgrade --force-reinstall --ignore-installed pip
that fixed the issue

This just WORKED

Answer 68 · 2020-04-08T11:20:12.000Z

Current version of pip3 for Ubuntu 18.04 arm64 is too low(9.0.1), so I upgrade it with pip3 install -U pip and ran into this problem. If installed by get-pip.py, then other problems will be caused, which is annoying.

Answer 69 · 2020-04-14T20:48:52.000Z

I just got this issue on a brand new install of Pycharm on Windows 10, with no prior install of python on the system.

Steps to reproduce:
Uninstall all python installs and ide's. Assure system is clean.
Reboot
install newest python - should be newest
install newest pycharm - Reproduced with 2020.1

Any attempt to update pip in the IDE will give this error:

Error: Traceback (most recent call last):
File "C:\Program Files\JetBrains\PyCharm 2020.1\plugins\python\helpers\packaging_tool.py", line 73, in run_pip
runpy.run_module(module_name, run_name='main', alter_sys=True)
File "G:\ProgramFiles_x86\Python38-32\lib\runpy.py", line 206, in run_module
return _run_module_code(code, init_globals, run_name, mod_spec)
File "G:\ProgramFiles_x86\Python38-32\lib\runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "G:\ProgramFiles_x86\Python38-32\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "G:\Duane\Dropbox\Projects\login_system\flaskdash\venv\lib\site-packages\pip_main.py", line 16, in
from pip._internal import main as _main # isort:skip # noqa
ModuleNotFoundError: No module named 'pip._internal'

So really I feel like the above blames users unfairly; Its also a bug in pycharm/pip that is doing this for a lot of people, IMHO. I cant see any other option existing because I just duplicated this multiple times in a clean VM with a brand new python install on it.

Answer 70 · 2020-04-14T21:14:53.000Z

So really I feel like the above blames users unfairly; Its also a bug in pycharm/pip

Agreed, this looks like a bug in pycharm (notice that the traceback refers to a pycharm helper). I'd suggest taking the issue up with them.

Answer 71 · 2020-04-14T21:21:45.000Z

No its a bug in pip too, but its made worse by pycharm's terrible design choices, IMHO.

Still, blaming the user is the wrong thing to do and I vote that you soften your tone as this defect reproduces with brand new systems that never had any python install on them before the issue is created by simply installing a standard python development environment on a clean system..

Answer 72 · 2020-04-14T21:27:57.000Z

Please demonstrate the pip bug with a clean install of pip not involving pycharm. I'm happy to address pip bugs, but we can't control when other tools use pip in ways that are explicitly unsupported (in this case, calling pip's internal functions).

Answer 73 · 2020-04-14T21:31:48.000Z

You don't seem to be understanding me, I'm not blaming the pip devs for anything except the dismissive and rude tone they are taking with users.

I'm saying this entire thread is conflating multiple issues and blaming multiple problems on the same thing when that's not the truth, and the error you are listing can actually be gotten multiple ways for multiple reasons.

You seem to be hungup ion the idea that its a user error; Its not. Its a dev error in pip and in any system that wants to use it, many users of pip never actually touch pip.

Answer 74 · 2020-04-14T22:13:07.000Z

@duaneking It's a dev error but not from the pip devs. If devs of software A write it in a way relies on internals of library B then devs of software A are to blame. The leading underscore _ should be sufficient indication for "use at your own risk". So you indeed seem to be blaming wrong devs.

Answer 75 · 2020-04-15T08:33:55.000Z

... and to be 100% clear, this issue is intended as a thread to help users understand and debug the various complex interactions around user config, system, 3rd party and pip behaviour that result in this type of error. The intention is to help users be able to identify where the problem lies, precisely so that they can direct issue reports at the right people.

If as a result of the diagnosis, a user confirms that there's a pip issue, then we'd welcome a new issue being raised, with a good description of the problem, and a means of reproducing it that we can replicate with just pip.

But I will say that my experience, and the evidence in this thread, is that very often the problem turns out not to be with pip, but with some other part of the "stack" - sorry if I come across as dismissive as a result, but please understand that I, and most of the other pip devs, really have seen these issues before. In the case of pycharm, I raised a similar issue with them about their not using supported means of installing packages, a long time (years) ago - to my knowledge it never got fixed, unfortunately. Hence my expectation in your case that it's a pycharm problem, and the traceback supports this.

Answer 76 · 2020-04-15T16:21:08.000Z

Again, you seem to be putting words in my mouth or misunderstanding me intentionally.

I'm pointing out issues for others. Your saying I'm saying something I'm not. Is English your first language? I'm happy to translate or find a way to make myself clearer.

There are multiple issues that get these errors.

I'm not blaming pip devs for pycharm's/jetbrains broken code.

I am blaming JetBrains devs for not actively working as software craftspeople and actively working to mitigate these issues by doing things like add workarounds to unblock people, or even just better error messages. But If the error path results are known, why not add some debugging and diagnostics or at least a message that links to this thread?

Also, A lot of people who get these errors never themselves touch pip, and thus logically they can't be at fault, so the dismissive tone is at best rude, and at worst targeting the wrong people and making them a victim of anger about a problem they are also innocent victims of.

Jetbrains Resharper uses a bad way to manage a virtualenv and last I checked uses its own packaged tools to do some of that management; This is why MacOS virtual environments are not compatible with numpty and other things by default, why you have these issues, because Pycharm uses its own package of a tool it calls venv instead of the virtual environment toolset that should be standard. The user is never told this and so the user rightly assumes that pip is at fault when in reality you can get this exact same error trying to install things that are not pip, without even touching pip.

Should Jetbrains be a more responsible member of the python community? Yes, Absolutely. But the python community could also do more to keep them in check, and can do more to make the common errors cited in this thread a mitigated problem that can more easily be worked around or could include some form of official guidance, and that's not happening either.

Answer 77 · 2020-04-17T17:47:56.000Z

I think a better solution is for pip to remain update agnostic. If distros won't keep pip up to date with their packages then we must force a newer version upon that distro when running pip3 install -U pip. I seriously doubt overwriting the pip installed with a distro with a newer version will break any python dependencies a distro is relying on. One of the great benefits of pip (along with npm and others) is that you can update your packages/dependencies agnostic of distro package updates. So you can actually keep your software updated without reliance on distro package maintainers. Instead, you get updates directly from PyPi from the python package maintainers.

Answer 78 · 2020-04-21T13:18:02.000Z

I would add this advice of not upgrading your system pip in the documentation.
I just followed these instructions and killed my raspbian pip and pip3 installs.

On the page:
https://packaging.python.org/tutorials/installing-packages/#ensure-pip-setuptools-and-wheel-are-up-to-date it lists python -m pip install --upgrade pip setuptools wheel , which is in conflict with recommentation 1 and 3 in this issue.

I think I restored that with:
python3 -m pip uninstall pip setuptools wheel
sudo apt-get --reinstall install  python3-setuptools python3-wheel python3-pip

python -m pip uninstall pip setuptools wheel
sudo apt-get --reinstall install python-setuptools python-wheel python-pip 
After that my raspbian PI zero pip and pip3 report the ancient 9.0.1 versions, which is the latests available according to https://tracker.debian.org/pkg/python-pip
The request to update pip in debian repo can be found here:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=901393

I did that, now I can't use pip3 anymore.
pip3 --version gives me:
-bash: /home/ubuntu/.local/bin/pip3: No such file or directory

I would be glad for any advice :)

Answer 79 · 2020-04-25T18:14:15.000Z

@duaneking

I am blaming pip devs for not actively working as software craftspeople and actively working to mitigate these issues by doing things like add workarounds to unblock people

No. If pip is broken, it needs to be fixed. However if a user is using it wrong, there is nothing that needs to be done. You dont add workarounds to fix people using software incorrectly, you educate those users on how to properly use the software. In this case, that was attempted via an issue raised with jetbrains. The fact that the user (jetbrains) didn't fix it is not pip's problem.

Answer 80 · 2020-05-02T02:09:05.000Z

I getting this error after failing to create a virtual environment in Pycharm and running the requirements.txt. The description above is way too technical for me to understand and sounds risky.
is There an easier way to handle this problem. I don't mind unistalling pip, if that can be done.

Answer 81 · 2020-05-09T11:58:50.000Z

This is my personal solution: I'm running Ubuntu 18.04 which comes with python 3.6.7.

I notice that pip is now included inside python, and it can be easily run by running python3 -m pip, so all I did is to create an alias in my .bashrc (or .zshrc file in my case) for pip:

alias pip='python3 -m pip'

Reload the shell and you are done!
Note: You don't need to install the package python3-pip, which is, by the way, very outdated.

Answer 82 · 2020-05-09T13:17:43.000Z

@Akronix, please elaborate what you mean by

I notice that pip is now included inside python

because that has no correlation with

it can be easily run by running python3 -m pip

python3 -m pip should work if pip can be found in one of the sites (usually [~/.local]/lib/python*/site-packages and AFAIK pip is only bundled with Python if one install it from the python.org's package. The package python3-pip from your is one of the providers, but I suspect that you had also installed pip from other sources, very likely to be PyPI. You should be able to inspect it a bit further by running python3 -m pip show pip.

The reason I make this comment is because I don't want you and others to have the confusion that pip is somehow part of CPython standard library. I am nothing against the alias and I want to leave a side note that keeping python3-pip might be a good idea just in case one run pip uninstall pip by mistake (like I did quite a few times, I'm not the most careful person ever).

Answer 83 · 2020-05-09T18:23:57.000Z

@Akronix, please elaborate what you mean by

I notice that pip is now included inside python

because that has no correlation with

it can be easily run by running python3 -m pip

python3 -m pip should work if pip can be found in one of the sites (usually [~/.local]/lib/python*/site-packages and AFAIK pip is only bundled with Python if one install it from the python.org's package. The package python3-pip from your is one of the providers, but I suspect that you had also installed pip from other sources, very likely to be PyPI. You should be able to inspect it a bit further by running python3 -m pip show pip.

The reason I make this comment is because I don't want you and others to have the confusion that pip is somehow part of CPython standard library. I am nothing against the alias and I want to leave a side note that keeping python3-pip might be a good idea just in case one run pip uninstall pip by mistake (like I did quite a few times, I'm not the most careful person ever).

oh, it might be the case, that I installed pip from PyPA and I completely forgot about it. Thank you for correcting me @McSinyx .

Answer 84 · 2020-05-14T16:12:51.000Z

I have fixed my pip on Fedora 32 by running sudo rm /usr/local/bin/pip

Answer 85 · 2020-05-19T19:47:25.000Z

I fixed this problem by reinstalling pip from get-pip.py

Not sure how it happened but I downloaded pip https://bootstrap.pypa.io/get-pip.py

the command to download is: curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
and executed the command python get-pip.py to reinstall pip.

Answer 86 · 2020-05-27T02:14:10.000Z

C:\Users\Kiddy>pip install virtalenv
WARNING: pip is being invoked by an old script wrapper. This will fail in a futu
re version of pip.
Please see #5599 for advice on fixing the unde
rlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip
directly.
ERROR: Exception:
Traceback (most recent call last):
File "C:\Users\Kiddy\AppData\Roaming\Python\Python37\site-packages\pip_intern
al\cli\base_command.py", line 186, in _main
status = self.run(options, args)
File "C:\Users\Kiddy\AppData\Roaming\Python\Python37\site-packages\pip_intern
al\commands\install.py", line 258, in run
isolated_mode=options.isolated_mode,
File "C:\Users\Kiddy\AppData\Roaming\Python\Python37\site-packages\pip_intern
al\commands\install.py", line 604, in decide_user_install
if site_packages_writable(root=root_path, isolated=isolated_mode):
File "C:\Users\Kiddy\AppData\Roaming\Python\Python37\site-packages\pip_intern
al\commands\install.py", line 549, in site_packages_writable
test_writable_dir(d) for d in set(get_lib_location_guesses(**kwargs))
File "C:\Users\Kiddy\AppData\Roaming\Python\Python37\site-packages\pip_intern
al\commands\install.py", line 549, in
test_writable_dir(d) for d in set(get_lib_location_guesses(**kwargs))
File "C:\Users\Kiddy\AppData\Roaming\Python\Python37\site-packages\pip_intern
al\utils\filesystem.py", line 140, in test_writable_dir
return _test_writable_dir_win(path)
File "C:\Users\Kiddy\AppData\Roaming\Python\Python37\site-packages\pip_intern
al\utils\filesystem.py", line 153, in _test_writable_dir_win
fd = os.open(file, os.O_RDWR | os.O_CREAT | os.O_EXCL)
PermissionError: [Errno 13] Permission denied: 'c:\program files\python37\Lib
\site-packages\accesstest_deleteme_fishfingers_custard_dhjxxo'
WARNING: You are using pip version 20.0.2; however, version 20.1.1 is available.

You should consider upgrading via the 'c:\program files\python37\python.exe -m p
ip install --upgrade pip' command.

C:\Users\Kiddy>

Answer 87 · 2020-05-27T02:14:35.000Z

what do you think I should do?

Answer 88 · 2020-05-27T04:19:37.000Z

@tootiefrutie9999, the error seems unrelated to the warning. I don't use Windows but I think you don't have permission to write to c:\program files\python37\Lib\site-packages\. I wonder why pip didn't fallback to user-site in your case though.

Answer 89 · 2020-05-29T13:19:01.000Z

C:\Users\Kiddy>path
PATH=C:\Program Files\Python37\Scripts;C:\Program Files\Python37;C:\Windows\sy
stem32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell
\v1.0;C:\Users\Kiddy\anaconda3;C:\Users\Kiddy\anaconda3\Library\mingw-w64\bin;C
:\Users\Kiddy\anaconda3\Library\usr\bin;C:\Users\Kiddy\anaconda3\Library\bin;C:
Users\Kiddy\anaconda3\Scripts;C:/Users/Kiddy/anaconda3/python.exe;C:/Users/Kiddy
/anaconda3/Scripts/conda.exe;C:\Users\Kiddy\AppData\Local\Programs\Microsoft VS
Code\bin;C:\Users\Kiddy\AppData\Local\Programs\Microsoft Code OSS\bin

C:\Users\Kiddy>

Answer 90 · 2020-05-29T13:19:16.000Z

this is my path

Answer 91 · 2020-05-30T14:05:46.000Z

I get same error on Linux, Ubuntu 18.04:

$ pip3 -V
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see #5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
pip 20.1.1 from /home/aivanf/.local/lib/python3.6/site-packages/pip (python 3.6)

How can I fix this?

More details: I installed pip using sudo apt-get install python3-pip because though I have Py3.6, it didn't have included pip package.

Answer 92 · 2020-05-30T14:08:36.000Z

@AivanF it's a warning, not an error. If it doesn't fail afterwards, I think you can live with it.

Answer 93 · 2020-06-05T18:07:01.000Z

Same problem on Windows,

$ pip -V
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see #5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
pip 20.1.1 from C:\Users\Gigacee\AppData\Roaming\Python\Python38\site-packages\pip (python 3.8)

Answer 94 · 2020-06-06T03:12:19.000Z

Hi @gigacee, on Windows it is recommended to invoke pip using python -m pip, as shown in the warning.

Answer 95 · 2020-06-06T14:09:03.000Z

Oh. I'm sorry, I mistakenly thought it was something of a bug. I rewrote the function in .bashrc, and the warning is no longer there. Thank you!

function pipclean() {
	pip list --outdated --format=freeze | awk -F "=" '{print $1}' | xargs pip install -U pip
}

↓

function pipclean() {
	python -m pip list --outdated --format=freeze | awk -F "=" '{print $1}' | xargs python -m pip install -U pip
}

Answer 96 · 2020-06-12T08:40:36.000Z

Hi @gigacee, on Windows it is recommended to invoke pip using python -m pip, as shown in the warning.

I am used to use pip. I wonder if there is a way to suppress this warnning ?

Answer 97 · 2020-06-12T09:39:57.000Z

@youngquan, I don't use Windows but I think you may find some sort of alias useful.

Answer 98 · 2020-06-22T10:25:52.000Z

Adding to this, I am on Windows so that package manager point probably does not apply.

C:\Users\aadibajpai>python -m pip
Traceback (most recent call last):
  File "C:\Users\aadibajpai\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "C:\Users\aadibajpai\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 142, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "C:\Users\aadibajpai\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "C:\Users\aadibajpai\AppData\Local\Programs\Python\Python38\lib\site-packages\pip\__init__.py", line 1, in <module>
    from pip._internal.utils.typing import MYPY_CHECK_RUNNING
  File "C:\Users\aadibajpai\AppData\Local\Programs\Python\Python38\lib\site-packages\pip\_internal\__init__.py", line 1, in <module>
    import pip._internal.utils.inject_securetransport  # noqa
ModuleNotFoundError: No module named 'pip._internal.utils'

Not sure how to proceed.

Answer 99 · 2020-07-01T12:03:03.000Z

@abhiTronix, I did rm -rf ~/.local/lib/python2.7/site-packages/
I can not see interpreter for python2.

Any way to return it before going format my PC.

Answer 100 · 2020-07-11T19:50:20.000Z

Never used pip. I had Anaconda installed and all seemed fine until conda broke (lots of complains about that online, no solution). Removed Anaconda, decided to try pip and installed it in Lubuntu 18.04.4:

$ sudo apt install python3-venv python3-pip

(/usr/bin/pip3 is created by the installation process.)

I followed the suggestion to upgrade

$ pip3 install -U pip

which installed pip-20.1.1 and broke pip, as described by pfmoore. Now I have, in addition to the /usr/bin/pip3 script, 3 python scripts at ~/.local.bin:

pip
pip3
pip3.6

Running pip or pip3.6, I get no errors or warnings. pip3 (the local bash script) still calls /usr/bin/pip3 with the resulting "WARNING: pip is being invoked by an old script wrapper..." All three scripts call, in the end, the new pip version.

The local pip3 calls /usr/bin/pip3, while the other two scripts don't. The 3 newer python scripts are identical, except for the filename. Apparently something in the environment (python's or, perhaps, bash's) is causing this behavior.

Decided to just delete (after renaming for tests) both pip3 scripts (local and at /usr/bin). Renaming only /usr/bin/pip3 does not solve the issue (pip3 would generate a 'file not found error). I hope this "solution" does not break something else in my system. My concern is that programs that call pip3 won't work.

I am keeping tabs of what is installed using pip and, if I suspect anything else is wrong, just uninstall it all, including, at the end, pip. Can always reinstall the Ubuntu package (with no upgrade).

If you have more than one script installed with new pip version, using the ones with names different from the older version avoids the warning. Renaming the offending local script to something else also works.

If anyone knows how to "repair" the environment to get rid of any mentions to "pip3", I thank you in advance.

Thanks to pfmoore and all the editors and commenters.

Luiz

PS: All working with latest (non-package) version of pip:

(1) deleted ~/local/bin/pip3 and /usr/bin/pip3, the offending scripts (the one at /usr was a leftover from the older, package version of pip)

(2) created link named pip3, just in case another program requires the pip3 command:
$ ln -s ~/.local/bin/pip ~/.local/bin/pip3