Whitespace in root path of virtualenv breaks scripts
vbabiy opened this issue Β· 59 comments
I'm not really sure if this is a distribute/setuptools/virtualenv but,
If I install virtualenv in
/var/lib/hudson/home/jobs/Minification WebHelpers/workspace/python/2.4
then run ./bin/easy_install:
bash: ./bin/easy_install: "/var/lib/hudson/home/jobs/Minification: bad interpreter: No such file or directory
Seems like something does not obey whitespace in path names correctly.
- Bitbucket: https://bitbucket.org/ianb/virtualenv/issue/38
- Originally Reported By: Domen KoΕΎar
- Originally Created At: 2010-05-07 14:57:32
+1, confirmed with Mac OS X 10.7.3 and Python 2.7.1
Kind of annoying, would be great to have a fix
We are able to create a virtualenv with spaces in the name (see #278), but easy_install and pip stumble over it later:
% virtualenv "foo bar"
New python executable in foo bar/bin/python
Installing setuptools............done.
Installing pip...............done.
% ./foo\ bar/bin/easy_install nose
zsh: ./foo bar/bin/easy_install: bad interpreter: "/tmp/cfl/foo: no such file or directory
127 % ./foo\ bar/bin/pip install nose
zsh: ./foo bar/bin/pip: bad interpreter: "/tmp/cfl/foo: no such file or directory
I'm also here to confirm that this is an issue with OS X (10.8 here). If you edit the VIRTUAL_ENV variable and shebangs in bin you can get it to work, but a fresh env chokes on any spaces in a path. Which is a big problem for OS X, given that the boot drive is typically named "Macintosh HD," so every path starts with "/Volumes/Macintosh HD..."
The hack I'm using works as follows.
bin/activate:
VIRTUAL_ENV='/Volumes/Macintosh\ HD/path/to/my/project'
bin/pip and bin/easy_install:
#!"/Volumes/Macintosh\ HD/path/to/my/project/venv/bin/python"
Pip seems to be working after escaping the space in the path.
Why was this closed? It's still very much an issue. edit my mistake, it's still open
this issue still shows open.
I was able to get around this my creating a symbolic link from my home directory to the one I was wanting to work in (that otherwise had a space in it).
I'm seeing this too because Mac. I get around this by manually editing the shebang line in the scripts to !#/usr/bin/env python and all works. However, as others have mentioned, this has to be done with each new env and any additional scripts installed in the env.
Seems this should be an easy fix in the code to either escape the space or use /usr/bin/env if is_darwin. However since I'm pretty much a noob at this I could be wrong.
This isn't just for mac, it's basically part of the specification/behaviour of *nix systems.
You can't have spaces in the first argument of the shebang line (they will get turned into separate arguments instead), and normally there's no escaping / quoting allowed either.
http://lists.gnu.org/archive/html/bug-bash/2008-05/msg00053.html
I know, I ran into this problem with anaconda as well. t's just endemic with Mac because the drive name has the whitespace in it.
It looks like this would be corrected by #611. Has there been any review of the efficacy of that pull request?
So annoying, should be fixed asap.
See the link @Ivoz posted, this is a Unix limitation. #611 might work for some Unix variants, if they support backslash escapes in a shebang line, but it's not clear which versions do (and the code just blindly does it without checking - which admittedly won't make the problem worse, but won't help either if it's not supported...)
It's indeed true that this is a result of the way that unix handles shebang lines, but if #611 fixes the problem for some systems and doesn't worsen the problem for others, would that still be an improvement?
If that's true then yes. But I can'rt comment on #611 as I'm not a Unix developer. There might be cases where it makes things worse, I just don't know. Sorry I can't be more help.
Fair enough. #611 probably needs to be more carefully looked at and tested for fringe cases.
Even worse: on windows it breaks on the default Jenkins path with the same error:
FATAL: whitespaces are not allowed in Python interpreter's path: C:\Program Files (x86)\Jenkins\shiningpanda\jobs\c3418983\virtualenvs\d41d8cd9
I was just affected by this issue. Following the instructions I found on StackOverflow, I managed to make pip work by just setting the first line to #!/usr/bin/env python
I'm not sure, however, if that solution works for all cases... I mean, I'm not sure about which Python will be executed
Changing the shebang of installed scripts to βenv pythonβ means that they will work only in an activated virtualenv. The scripts were generated with explicit absolute paths so that they would always use the Python in the venv, and thus find the installed packages needed by the scripts.
My suggestion would be that someone (probably someone affected by this issue, but at a minimum someone on a platform that has the problem and also has a way of solving it) provide a pull request implementing a check along the lines of:
- If we have spaces in the pathname,
- and we are on platform XXX,
- then write the shebang line with the following escaping to handle the spaces.
- In all other cases, fall back to the current behaviour.
Further additions could then be made by interested parties just adding extra platform checks.
Ideally, it would be nice to include a comment with a link to the documentation that confirms how platform XXX supports paths with spaces, so that future maintainers have a reference to check against. Personally, I'm not clear what fixes work and where:
- the discussion here suggests that double quotes work on OSX, but does that depend on the precise OSX version?
- In #611 escaping spaces with backslashes was used, but there was no confirmation as to what OS it was for (Linux? A specific kernel version? A specific distro?)
Note that no such platform-specific variants should use /usr/bin/env. As @merwok pointed out, that results in a change in behaviour - the shebang is deliberately written to allow running the script without activating the environment.
Adding some tests to be sure the behaviour is as expected (including the principle that it falls back when we're not on a specifically recognised platform) would be extremely useful, too, but it would also be fiddly, as it would involve monkeypatching to allow testing for platform XXX when you're not actually running on that platform.
@pfmoore As I mentioned, I was recently affected by this issue and I'm running Linux Mint 18. I have never contributed with Virtualenv but I am currently at Python Brasil and we'll have a day dedicated to sprints, I might give it a try!
escaping with backslashes or quotes won't work, according to https://lists.gnu.org/archive/html/bug-bash/2008-05/msg00052.html
Experimentally, I can verify that escaping with backslashes or quotes doesn't work with OSX 10.11.6.
virtualenv should stay away from kernel-dependent shebangs. I've checked Linux, XNU (macOS's kernel), FreeBSD, OpenBSD and NetBSD source codes. None of them can handle spaces in shebang.
Before there's a fix, don't use spaces.
I submit a patch that adds a warning for Python 3's new venv, which is quite similar to virtualenv but rejected by @vsajip: http://bugs.python.org/issue28446. Indeed it's not Python's fault but an operating system's. Maybe this issue can be closed?
As an extra data point, note the behaviour on Windows, which seems to be as expected:
C:\Users\Vinay> "\Temp\aaa bbb\Scripts\pip" --version
pip 6.0.8 from C:\Temp\aaa bbb\lib\site-packages (python 3.4)
C:\Users\Vinay> pyzzer -i "\Temp\aaa bbb\Scripts\pip.exe"
There is a launcher.
Shebang: #!"C:\Temp\aaa bbb\Scripts\python.exe"
Archive contents:
__main__.py
C:\Users\Vinay> "\Temp\aaa bbb\Scripts\python" -m pip install -U pip
You are using pip version 6.0.8, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pip from https://pypi.python.org/[...]/pip-9.0.1-py2.py3-none-any.whl#md5=297[...]
Using cached pip-9.0.1-py2.py3-none-any.whl
Installing collected packages: pip
Found existing installation: pip 6.0.8
Uninstalling pip-6.0.8:
Successfully uninstalled pip-6.0.8
Successfully installed pip-9.0.1
C:\Users\Vinay> "\Temp\aaa bbb\Scripts\pip" --version
pip 9.0.1 from C:\Temp\aaa bbb\lib\site-packages (python 3.4)
Yep virtualenv scripts on Windows work as distlib defines its own shebang protocol in PC/launcher.c. Maybe POSIX can have something similar - a user space shebang parser instead of unreliable kernels.
a user space shebang parser instead of unreliable kernels
I'm not sure why Bash can't do this (for example) - I don't think it's a kernel-space thing.
Shebangs are handled in kernel space because it should be usable outside shells.
Technical details:
On UNIX-like systems (Linux, Mac, *BSD, ...), a new program is created via fork() and exec(). exec() is similar to CreateProcess() on Windows, which runs a new program. On UNIX-like systems, exec() eventually calls the system call execve(). The latter function is implemented in kernels, so shebang parsing is done in kernels.
It can't be implemented in C libraries, either, or static linked programs or programs that uses system calls directly (via int 80 or sysenter, etc.) won't work.
Maybe we should just forbid the creation of a virtual environment with spaces in the path. It won't work as expected anyway
Shebangs are handled in kernel space because it should be usable outside shells
Yes, but couldn't a shell do the parsing itself if the system call returned ENOEXEC? I realise it might be a can of worms ...
Interesting tidbit - the kernel functionality on Linux appears to have been written by long-time Python committer Martin von LΓΆwis :-)
This page was also an interesting read: http://www.in-ulm.de/~mascheck/various/shebang/
Yes, but couldn't a shell do the parsing itself if the system call returned
ENOEXEC?
IMO different semantics between shells and the underlying kernel would bring lots of confusion to users as well as developers. Currently at least bash and zsh do parsing when execve() fails, but only for better error reporting, not providing a fallback.
Interesting tidbit - the kernel functionality on Linux appears to have been written by long-time Python committer Martin von LΓΆwis :-)
Interesting! I didn't notice that :) Also thanks for the extra material. While reading the kernel source code is the fastest way, such documents is still helpful.
About @fbidu's idea:
Maybe we should just forbid the creation of a virtual environment with spaces in the path. It won't work as expected anyway
Creating virtual environments with fragile paths are useful for testing corner cases in path handling. In this example it demonstrates how kernels are broken. My idea is adding warnings instead of forbidding, just like the patch I've posted to http://bugs.python.org/issue28446
I think #994 "Pip fails with space in virtualenv path" is a duplicate of this issue.
I want to repeat the comment from #997 (comment), "virtualenv are broken with fragile kernel shebang parsing." And in that spirit, #1014 "Not compatible with a directory having emojis in its path" is another example of virtualenv being broken by fragile kernel shebang parsing. I'll bet the problem occurs with any non-ASCII characters in the path, in fact.
Maybe we should collect all three aspects of fragile kernel shebang parsing into one issue, so that we can be sure that one fix can address spaces, length, and non-ASCII characters in the virtualenv path? I nominate this issue, because it's the oldest.
While we work on a fix, I think it would be good for virtualenv to print a warning when asked to create an environment in a path which a) has spaces, b) is too long, or c) has non-ASCII characters. A sentence in some documentation would help also.
Hi all,
This is super annoying and such a huge waste of time due to something seemingly so simple (simple cause at least).
How about renaming (when possible), or using links (creating a directory such as /virtualenvs/python3.5 without space, then letting this be a soft link to the original directory?)
creating a directory such as /virtualenvs/python3.5 without space
Another project virtualenvwrapper has done something quite similar.
something seemingly so simple (simple cause at least).
Neither are simple. The cause is related to kernel parsing codes and a workaround requires user space shebang handling.
It's still an issue 6 years later?
This issue tripped me up 2 weeks ago. Yes, it's still an issue.
Note that as it's a Unix issue rather than a virtualenv issue, it's unlikely that it'll get "fixed" unless the Unix kernel limitation is removed...
The first virtualenv bug is that virtualenv chose to implement shell-executable files by using a certain Unix kernel feature, despite that feature having limitations that routinely cause problems for virtualenv user. Virtualenv could fix this by using a different mechanism for shell-executable files. The second virtualenv bug is to have no documentation of how users should use virtualenv in order to work around the first bug (create a virtualenv only on short paths with ASCII-only characters and no whitespace). The third virtualenv bug is that it has no mechanism which detects cases where the user chooses a virtualenv path which will trigger the first bug, and print a helpful warning message. There's plenty that virtualenv contributers could do to improve the situation.
@JDLH About your first "bug": it's not handled by virtualenv but distlib, and there's an implementation at https://bitbucket.org/pypa/distlib/pull-requests/31/. I hope I could have time to study the internals of distlib and reply to Vinay Sajip's comments properly but unfortunately I don't.
The first virtualenv bug is that virtualenv chose to implement shell-executable files by using a certain Unix kernel feature
@JDLH This is not specific to virtualenv - all Unix script files (i.e. files with shebang lines) use this kernel feature - and there is no compelling reason for virtualenv (or anything else) to reinvent a completely new method of dispatching scripts, when the existing mechanism is so widely used and well understood. If you wrote a script by hand which had spaces in the interpreter path (no virtualenv involved) it would exhibit the same problem.
There's plenty that virtualenv contributers could do to improve the situation.
There are probably many calls on contributors' time. This issue affects the relatively small number of cases where long paths/paths with spaces are used. Perhaps some of those users who are affected could help the contributors to help them by proposing a patch which would help with the detection and warning messages? Just an idea.
@yan12125 distlib allows one to specify the executable in the shebang line however one wants. The kernel limitation on whitespace / long lines perhaps won't ever be fixed by Linux developers because of backward compatibility. One could just provide a custom string for the shebang executable (incorporating a generalised equivalent to the Mozilla script hack) and distlib should write it into the script, so one can experiment with it just as a distlib user (hence with no need to look at the internals, AFAICT).
This is not specific to virtualenv
@vsajip This is a true statement β other software uses the same shebang mechanism which virtualenv chose β but it misses the point of this issue. There is nothing about the value provided by virtualenv that demands it use the shebang. virtualenv could use a different mechanism, but it chose the shebang, and thus virtualenv really does inherit the shebang's limitation.
there is no compelling reason for virtualenv (or anything else) to reinvent a completely new method of dispatching scripts
I think what you are saying is that the problems experienced by the people who have encountered this issue over the years are not "compelling". I think the reason why so many people find and comment on this issue over the years is that they do find the problems "compelling". I certainly do.
Perhaps some of those users who are affected could help the contributors to help them by proposing a patch which would help with the detection and warning messages? Just an idea.
I chose the word "contributors" to include both the stalwarts who do most of the work on virtualenv, and the visitors like me who find this problem compelling enough to work on a reducing its impact. It's fair to say that we who find the problem compelling should contribute a patch.
It would be helpful if the stalwarts who know virtualenv well could suggest promising approaches. If I wanted to insert a warning message for the user who types virtualenv ~/my\ long\ path\ with\ spaces, where should that code best reside? Are there non-obvious constraints on such a patch, which the stalwarts could share, to remove an obstacle from the visitor working on their first contribution? Do the stalwarts have some historical objection to accepting such a patch? (I mean, I can't be the first person who thought of adding a warning message. There has to be a reason it hasn't happened yet.)
There is one well-known path that contains spaces and cannot be modified: /Users/iulian/Library/Mobile Documents/com~apple~CloudDocs. So anyone who wants to keep some scripts managed with virtualenv in iCloud bumps into this problem.
It would be helpful if the stalwarts who know virtualenv well could suggest promising approaches.
Well, if we knew any, we would probably not have left this issue unresolved for so long, given the amount of criticism we seem to get for doing so :-(
If I wanted to insert a warning message for the user who types virtualenv ~/my\ long\ path\ with\ spaces, where should that code best reside?
You could start by looking at the argument parsing code, adding a check once the path has been determined.
Are there non-obvious constraints on such a patch, which the stalwarts could share, to remove an obstacle from the visitor working on their first contribution?
They can mostly be found by searching the issue list here for the various comments made on this and other issues over the years, but to start with, you need to not reject paths when they will work - and that means working out what limitations the OS imposes. These vary drastically between systems. Windows allows spaces and long filenames, some Unix systems allow spaces, some need paths with spaces to be quoted, some have very short length limits (32 characters?) some longer, ... Many limitations aren't well documented, and very few contributors have access to sufficient systems to be able to test all the possibilities enough to supplement the available docs.
Do the stalwarts have some historical objection to accepting such a patch?
No, other than "don't assume it's as simple as you think at first glance, and don't ignore all of the various (sometimes pretty obscure) systems we have to support".
If someone does want to take a stab at this - and they should be aware that it's not something that I'd personally recommend as a "first contribution" - then they should start by reading all of the history available in the various issues (some linked from this one, others probably not, some probably on the pip tracker and maybe even the distlib or setuptools trackers) and summarise in a new PR the constraints imposed by various OSes. Propose that as a documentation patch that states "unless these conditions are met, the shebang headers used by virtualenv will not work as expected, and so virtualenv does not support creating environments in directories that don't match the stated conditions". The PR can include a code change to warn if the documented conditions are not met, or can propose this as future work (as from what I recall, it's pretty hard to introspect the system details precisely enough to know what limits apply - consider "Ubuntu with a patched kernel"...). Personally, I'd be OK with a warning that only flagged known cases where things would fail, and was silent if it wasn't sure. But I'd also be fine with a docs-only patch at this stage.
You'd then need to get reviews of your patch from people with access to the systems you cover - as noted, the core devs can't really help here because none of us use (for example) FreeBSD, or OpenBSD, or Solaris, or ...
You could also just do a partial job, and create a PR that added docs and a warning only for (say) OSX. I don't know if that would stop the complaints about this issue (I don't have a feel for what systems this comes up on most frequently) but maybe it would be sufficient. Possibly one of the core devs who uses OSX would be OK with merging that.
Does that help?
Perhaps I don't understand something, but couldn't this issue be solved by having (for example) bin/pip contain
#!/bin/sh
"/my/long path/with spaces/pythonx.y" "/path/to/my project/with spaces/venv/bin/real-pip" "$@"
and then moving the current pip script to real-pip? I don't see why we have to use the shebang directly.
@raxod502 I assume you're aware this wouldn't work on Windows. Also, I think the "normal" incantation needed on the second line is more complex than the one you give (although I don't know why, personally). You can probably find the proper approach via a web search. With your approach, what would happen for paths with " or $ characters in them?
Assuming you can address these sorts of issue, I guess the next step would be for you to submit a PR (as per the comments above) and we can debate specifics on that. You'd need to get at least one of the virtualenv core committers who works on Unix involved - as a Windows user I wouldn't be willing to merge a Unix-specific PR like this myself.
@gst Short answer, no. Long answer is at http://paul-moores-notes.readthedocs.io/en/latest/wrappers.html There have been many discussions on this point over the years, and every time someone has come up with a solution other than an exe wrapper, it's had problems.
In this case, please remember this problem does not exist on Windows. So there's absolutely no reason to change anything in the Windows environment - any change must be restricted to Unix only.
thx for good answer :)
other than an exe wrapper, it's had problems.
personally I can live with that (if I had to).
I've updated distlib to handle long paths and paths with spaces. I didn't use Harald Nordgren's patch directly (it had some problems - e.g. no tests) but the approach is the same - using '/bin/sh' as the executable. pip / virtualenv maintainers might want to test after locally vendoring the current version of the distlib repository.
The development version of pip now does vendor this newer version of distlib, which means pip now handles spaces in directory names just fine.
As I understand it, once the next release of pip and a virtualenv release are made, this issue will get fixed -- any new virtualenv created will support spaces in path, as will binaries installed by pip (except possibly in some quirky cases like setuptools' wrappers that are not directly installed by pip).
I've just started using gvfs to mount samba shares in user space and find virtualenv/pip etc scuppered on account of punction in the mount paths generated by gvfs.
There are no spaces in the paths, but plenty of other things to bring virtualenv/pip etc to their knees trying to run in a dir like
/run/user/1000/gvfs/smb-share:server=bolt,share=eng/projects/msp/mrfbus/land
As far as I can see there is no option currently in gvfs to prevent punctuation in the paths of it's generated mount points. My only workaround is to never create a virtualenv on a gvfs mount , which seems a little sad
@pradyunsg Is there any timeline for the next pip release? The last one was over a year ago, and it seems kinda dumb to wait so long for this really simple fix to show up in virtualenv.
Hi @raxod502!
Yeah, we want to get it out soon. The thing is we're short on developer time to get PEP 518 out the door, which is something we want to do. It might just happen that there's a pip 10 without PEP 518 but again, it depends on finding the time to get the plan settled on that.
It appears this bug is fixed by pip 10.0.0, released 2018-04-14. Strictly speaking the fix was in distlib 0.2.6, which was vendored into pip with their PR4819 in Oct 2017, which first appeared in pip 10.0.0b2.
The underlying fix appears to be in distlib's commit 9285cca. As vsajip commented here on May 28, 2017, the approach follows Harald Nordgren's idea: keep using a simple shebang for simple cases (OS is not Posix, or path is short enough and has no spaces), but for the non-simple cases use a /bin/sh exec command instead, which can handle the long or space-containing paths.
I did a quick test on Mac OS X 10.11.6, making a virtual env in a long path with spaces in it, then invoking pip3 install, and it seems to work. I haven't fully tested that everything described in this bug is fixed on every OS.
After upgrading pip from 9.0.1 to 18.1 (they switched to calendar based version) and virtualenv from 15.0.1 to 16.0.0 using Ubuntu 16.04.5 LTS the problem seems to be gone:
sudo pip install --upgrade pip
sudo pip install --upgrade virtualenv
Now all pip commands work properly in virtualenvs having spaces in their root path.
Old ticket: but why not
#!/usr/bin/env python ?
@rirl , I think leaving a comment on a closed Issue is unlikely to get your idea attention. The problem was given a solution. If you think that the new virtulenv, with this solution, would be improved by doing things differently, then you are proposing a new change. Consider opening a new Issue, say what change you think should happen, and make a case for why the change would be better than the new virtualenv.