sys.path will use user site-packages over the environment's site-packages
Opened this issue · 13 comments
Hello,
I'm following up on a recommendation from @jakirkham in the R-base feedstock in this issue:
conda-forge/r-base-feedstock#37 where we see similar behavior in R with respect to .libPaths().
According to PEP 370 there are platform-specific paths in which users can install packages which will be included in sys.path. The unfortunate result is that if you have a package installed there, it may be incompatible with a given conda environment's binaries and cause issues. Worse, the user-package is always preferred in import-order, so this will impact every environment (including root).
I'm kind of torn as to whether this is or is not a bug, but I'm reporting it in any case for further discussion.
Here is a demonstration showing this in a root environment from defaults and a conda-forge environment using the Python from this feedstock.
boring conda-info for completeness
16:33:55 ~
§ conda info
active environment : None
user config file : /home/evan/.condarc
populated config files :
conda version : 4.4.11
conda-build version : 3.0.28
python version : 3.6.1.final.0
base environment : /opt/Miniconda3 (read only)
channel URLs : https://repo.continuum.io/pkgs/main/linux-64
https://repo.continuum.io/pkgs/main/noarch
https://repo.continuum.io/pkgs/free/linux-64
https://repo.continuum.io/pkgs/free/noarch
https://repo.continuum.io/pkgs/r/linux-64
https://repo.continuum.io/pkgs/r/noarch
https://repo.continuum.io/pkgs/pro/linux-64
https://repo.continuum.io/pkgs/pro/noarch
package cache : /opt/Miniconda3/pkgs
/home/evan/.conda/pkgs
envs directories : /home/evan/.conda/envs
/opt/Miniconda3/envs
platform : linux-64
user-agent : conda/4.4.11 requests/2.14.2 CPython/3.6.1 Linux/4.15.7-1-ARCH arch/ glibc/2.26
UID:GID : 1000:100
netrc file : None
offline mode : False
Set up the user-site packages and add a Hello World package:
16:33:59 ~
§ mkdir -p ~/.local/lib/python3.6/site-packages
16:34:07 ~
§ echo "print('Hello World')" > ~/.local/lib/python3.6/site-packages/hello.py
Try it in the root environment:
16:34:13 ~
§ which python
/opt/Miniconda3/bin/python
16:34:20 ~
§ python
Python 3.6.1 |Continuum Analytics, Inc.| (default, May 11 2017, 13:09:58)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/opt/Miniconda3/lib/python36.zip', '/opt/Miniconda3/lib/python3.6', '/opt/Miniconda3/lib/python3.6/lib-dynload', '/home/evan/.local/lib/python3.6/site-packages', '/opt/Miniconda3/lib/python3.6/site-packages']
>>>
>>> import hello
Hello World
>>>
Note the end of sys.path: '/home/evan/.local/lib/python3.6/site-packages', '/opt/Miniconda3/lib/python3.6/site-packages'] the .local site-packages takes precedence over the environment's.
Try it in a fresh conda-forge environment:
16:34:40 ~
§ conda create -n example -c conda-forge python=3.6
Boring output of install plan
Solving environment: done
## Package Plan ##
environment location: /home/evan/.conda/envs/example
added / updated specs:
- python=3.6
The following packages will be downloaded:
package | build
---------------------------|-----------------
certifi-2018.1.18 | py36_0 143 KB conda-forge
setuptools-38.5.2 | py36_0 525 KB conda-forge
------------------------------------------------------------
Total: 668 KB
The following NEW packages will be INSTALLED:
ca-certificates: 2018.1.18-0 conda-forge
certifi: 2018.1.18-py36_0 conda-forge
ncurses: 5.9-10 conda-forge
openssl: 1.0.2n-0 conda-forge
pip: 9.0.1-py36_1 conda-forge
python: 3.6.4-0 conda-forge
readline: 7.0-0 conda-forge
setuptools: 38.5.2-py36_0 conda-forge
sqlite: 3.20.1-2 conda-forge
tk: 8.6.7-0 conda-forge
wheel: 0.30.0-py36_2 conda-forge
xz: 5.2.3-0 conda-forge
zlib: 1.2.11-0 conda-forge
Proceed ([y]/n)? y
Downloading and Extracting Packages
certifi 2018.1.18: ##################################################### | 100%
setuptools 38.5.2: ##################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use:
# > source activate example
#
# To deactivate an active environment, use:
# > source deactivate
#
Run the same test:
16:35:21 ~
§ source activate example
(example) 16:35:27 ~
§ which python
/home/evan/.conda/envs/example/bin/python
(example) 16:35:29 ~
§ python
Python 3.6.4 | packaged by conda-forge | (default, Dec 23 2017, 16:31:06)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/home/evan/.conda/envs/example/lib/python36.zip', '/home/evan/.conda/envs/example/lib/python3.6', '/home/evan/.conda/envs/example/lib/python3.6/lib-dynload', '/home/evan/.local/lib/python3.6/site-packages', '/home/evan/.conda/envs/example/lib/python3.6/site-packages']
>>>
>>> import hello
Hello World
>>>
Same behavior for .local site-packages as in root.
I believe this behavior is true for most (all?) Python installs, including those provides by various linux distributions, macPorts, brew, etc. Since the search path order is specified in PEP 370 "The user site directory is added before the system site directories but after Python's search paths and PYTHONPATH", I think it would be unwise to change this behavior.
This can cause problems if incompatible packages are installed into the user site-packages directory, but it can also be used as a feature to provide packages which are shared by all Python installs.
What if we added some kind of warning (maybe at startup) to point out .local is on the path? While I understand the rationale for the PEP, it does feel a little strange in the context of conda where users can easily create their own environments anywhere. So the value of looking in .local is at least significantly reduced if not problematic/at odds with typical use cases.
Does virtualenv do anything to disable .local?
Great question. Not sure.
So Christian Heimes, author PEP 370, is proposing it be deprecated.
Based on that conversation, it does sounds like pip is moving more towards using user-installs as a default. That could potentially put Python in a similar situation as R is today (where user-installs are very commonplace).
Yeah, it does sound like .local is here to stay.
If you have time, @ebolyen, I'd suggest researching @jjhelmus' question above. Namely what does virtualenv do with .local? Does it disable it? Does it make itself higher priority? Does it do nothing?
Once we have answers to these questions, we can see whether that solution would apply for us.
It looks like venv disables it:
11:22:02 ~
§ python3 -m venv --without-pip venv
11:22:17 ~
§ source venv/bin/activate
(venv) 11:23:05 ~
§ python
Python 3.6.1 |Continuum Analytics, Inc.| (default, May 11 2017, 13:09:58)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/opt/Miniconda3/lib/python36.zip', '/opt/Miniconda3/lib/python3.6', '/opt/Miniconda3/lib/python3.6/lib-dynload', '/home/evan/venv/lib/python3.6/site-packages']
>>>
(venv) 11:23:20 ~
§ ls ~/.local/lib/python3.6/site-packages/
hello.py __pycache__
This seems to also be corroborated in this (kind of related) issue about the --user flag of pip with venv: pypa/pip#4141
Is this documented behavior?
Not yet: pypa/virtualenv#1005?
But it does seem to be the running behavior for a while: minor note in changelog, related PR: pypa/virtualenv#803
Yeah, turns out I've been looking in the wrong repo. But good news! There's actually a PEP that defines this behavior:
PEP 405 specifically calls out how it interacts with PEP 370 in this section.
At this point it does seem like conda should behave (at least by default) the same way as Python's builtin venv module.
I found that in conda/conda#448 this issue was discussed in 2014. The decision at that time was that conda should not exclude .local. One helpful comment in that thread was that setting export PYTHONNOUSERSITE=True will ignore the local user site directory.
It may be worth re-opening that discussion if there is interest.
For reference, section of PEP 405 pointed out by @ebolyen above:
PEP 370 user-level site-packages are considered part of the system site-packages for venv purposes: they are not available from an isolated venv, but are available from an
include-system-site-packages = truevenv.