pip rewrites ipython shebang to python
Closed this issue · 6 comments
Environment
- pip version: 20.2
- Python version: 3.7
- OS: RedHat 7
Description
We have a custom python interpreter named "mpython" (similar to ipython). Users write scripts with a shebang line that says #!/usr/bin/env mpython and want to delivery them with pip. But when pip installs them into a virtual environment (python 3.7 venv library), it thinks that mpython is close enough to python that it rewrites the line to point at the at venv python. This happens with our interpreter (mpython) and ipython.
Before install:
#!/bin/env ipython
...
After install:
#!/path/to/venv/bin/python
...
Expected behavior
I'm assuming the regular expression for the shebang is too greedy. It is matching *python maybe. It would be nice if it could at least put application name back to the name that was in the script when it rewrites the line. So it would be nice if it did this:
#!/path/to/venv/bin/ipython
If that's not possible, maybe having a flag in setup.py to control the shebang rewrite would also work.
ps: This might be setuptools/distutils problem - I don't know how pip works internally that well. Feel free to point me to the right place to file this if that's the case.
Can you please provide instructions to reproduce this? For example, a setup.py and a set of commands to run and demonstrate the problem.
Sure - here's a repo that demonstrates the problem. The README has instructions:
https://github.com/TD22057/ipybug
python -m venv venv
source venv/bin/activate
pip install -U pip setuptools
pip install git+https://github.com/TD22057/ipybug
Look at venv/bin/ipy_script.py, the shebank line will be:
#!/path/to/venv/bin/python
print("Testing 1 2 3")
but the original script in the repository is this:
#!/bin/env ipython
print("Testing 1 2 3")
I don't know if this matters or not but I was trying to track this down by adding a bunch of log messages to _vendor/distlib but it looks like that's not being used in this case. Turning on verbose, I get a message saying it's running a python command to import setuptools and do a bunch of stuff. The actual install command in setuptools is using distutils, not distlib which comes out of my python 3.7 install. The reason I thought this might matter is that I see a message saying copying and adjusting ipy_script.py in the log and, thinking that must be where the problem is, I searched for it and found it in _vendor/distlib but that's not being run in this case.
It looks like (to me anyway) a bug in distutils/command/build_scripts.py. There is a regex that is '^#!.*python[0-9.]*([ \t].*)?$' that will match anything between #! and python. That makes distutils think it's python and it replaces the shebang line without looking at any other part of it. So basically if the shebang line ends in the string python, it's going to trigger.
Do you get the same behaviour if you install wheel in the venv before installing your package? If you don't have wheel installed, pip asks setuptools to do the install (via setup.py install) and it's setuptools that creates the wrappers. If you have wheel installed, pip builds a wheel (via setup.py bdist_wheel) and installs that - in that case it's pip's code that writes the wrappers.
Yes - that works. I expanded my test locally for every combination I can think of that a user might do to shebang ipython and they all fail with distutils. They all work fine (in the sense that the ipython shebang is left alone) if wheel is installed.
Closing this since it's not your problem. Thanks for the help.