canonical/pylxd

setup.py should not put runtime dependencies in setup_requires as it breaks pip dependency management

Closed this issue · 9 comments

pdecat commented

When installing pylxd with pip, pip invokes its setup.py with the egg_info command to retrieve egg metadata about pylxd (cf. https://pip.readthedocs.io/en/stable/reference/pip_install/#build-system-interface).

This has the side effect of installing pylxd dependencies which currently breaks because of an incompatibility between current requests 2.18.4 and urllib3 1.23.

Example with current tarball from https://pypi.org/project/pylxd/#files:

# wget -q https://files.pythonhosted.org/packages/a0/5e/af099af60d089b28df6b550d34c1e807ce4e3906d257744f55573c4d3cbb/pylxd-2.2.6.tar.gz                            
# tar xf pylxd-2.2.6.tar.gz
# cd pylxd-2.2.6/
# virtualenv test
New python executable in /home/pdecat/workspaces/tmp/pylxd-2.2.6/test/bin/python
Installing setuptools, pip, wheel...done.
# ./test/bin/pip freeze
# ./test/bin/python setup.py egg_info
Installed /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs/requests_unixsocket-0.1.5-py2.7.egg
Searching for requests!=2.8.0,>=2.5.2
Reading https://pypi.org/simple/requests/
Downloading https://files.pythonhosted.org/packages/49/df/50aa1999ab9bde74656c2919d9c0c085fd2b3775fd3eca826012bef76d8c/requests-2.18.4-py2.py3-none-any.whl#sha256=6a1b267aa90cac58ac3a765d067950e7dbbf75b1da07e895d1f594193a40a38b
Best match: requests 2.18.4
Processing requests-2.18.4-py2.py3-none-any.whl
Installing requests-2.18.4-py2.py3-none-any.whl to /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs
writing requirements to /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs/requests-2.18.4-py2.7.egg/EGG-INFO/requires.txt

Installed /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs/requests-2.18.4-py2.7.egg
Searching for pbr>=1.8
Reading https://pypi.org/simple/pbr/
Downloading https://files.pythonhosted.org/packages/b3/5d/c196041ffdf3e34ba206db6d61d1f893a75e1f3435699ade9bd65e089a3d/pbr-4.0.4-py2.py3-none-any.whl#sha256=3747c6f017f2dc099986c325239661948f9f5176f6880d9fdef164cb664cd665
Best match: pbr 4.0.4
Processing pbr-4.0.4-py2.py3-none-any.whl
Installing pbr-4.0.4-py2.py3-none-any.whl to /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs

Installed /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs/pbr-4.0.4-py2.7.egg
Searching for urllib3>=1.8
Reading https://pypi.org/simple/urllib3/
Downloading https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl#sha256=b5725a0bd4ba422ab0e66e89e030c806576753ea3ee08554382c14e685d117b5
Best match: urllib3 1.23
Processing urllib3-1.23-py2.py3-none-any.whl
Installing urllib3-1.23-py2.py3-none-any.whl to /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs
writing requirements to /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs/urllib3-1.23-py2.7.egg/EGG-INFO/requires.txt

Installed /home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs/urllib3-1.23-py2.7.egg
Traceback (most recent call last):
  File "setup.py", line 34, in <module>
    pbr=True)
  File "/home/pdecat/workspaces/tmp/pylxd-2.2.6/test/local/lib/python2.7/site-packages/setuptools/__init__.py", line 128, in setup
    _install_setup_requires(attrs)
  File "/home/pdecat/workspaces/tmp/pylxd-2.2.6/test/local/lib/python2.7/site-packages/setuptools/__init__.py", line 123, in _install_setup_requires
    dist.fetch_build_eggs(dist.setup_requires)
  File "/home/pdecat/workspaces/tmp/pylxd-2.2.6/test/local/lib/python2.7/site-packages/setuptools/dist.py", line 514, in fetch_build_eggs
    replace_conflicting=True,
  File "/home/pdecat/workspaces/tmp/pylxd-2.2.6/test/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 779, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (urllib3 1.23 (/home/pdecat/workspaces/tmp/pylxd-2.2.6/.eggs/urllib3-1.23-py2.7.egg), Requirement.parse('urllib3<1.23,>=1.21.1'), set(['requests']))

As mentioned in several places, runtime dependencies should not be put in setup_requires as it breaks pip's dependency management:

Warning Finally, beware of the setup_requires keyword arg in setup.py. The (rare) packages that use it will cause those dependencies to be downloaded by setuptools directly, skipping pip's protections. If you need to use such a package, see Controlling setup_requires.

cf. https://pip.readthedocs.io/en/stable/user_guide/#installation-bundles

Controlling setup_requires

Setuptools offers the setup_requires setup() keyword for specifying dependencies that need to be present in order for the setup.py script to run. Internally, Setuptools uses easy_install to fulfill these dependencies.

pip has no way to control how these dependencies are located. None of the Package Index Options have an effect.

cf. https://pip.readthedocs.io/en/stable/reference/pip_install/#controlling-setup-requires

Related:

pypa/pip#1884
pypa/pip#3691
pypa/pip#4156 (comment)

psf/requests#4669
psf/requests#4671

pdecat commented

To provide a bit more context, this currently prevents installing pylxd even when all its dependencies are pinned in a requirements.txt file:

# docker run -ti python:2.7 bash
root@df4a2f10d39a:/# cat > requirements-lxd.txt <<EOF
> asn1crypto==0.24.0        # via cryptography
> certifi==2018.4.16        # via requests
> cffi==1.11.5              # via cryptography
> chardet==3.0.4            # via requests
> cryptography==2.2.2       # via pylxd, pyopenssl
> enum34==1.1.6             # via cryptography
> idna==2.6                 # via cryptography, idna, requests
> ipaddress==1.0.22         # via cryptography
> pbr==4.0.4                # via pylxd
> pycparser==2.18           # via cffi
> pylxd==2.2.6
> pyopenssl==18.0.0         # via pylxd
> python-dateutil==2.7.3    # via pylxd
> requests-toolbelt==0.8.0  # via pylxd
> requests-unixsocket==0.1.5  # via pylxd
> requests==2.18.4          # via pylxd, requests-toolbelt, requests-unixsocket
> six==1.11.0               # via cryptography, pylxd, pyopenssl, python-dateutil
> urllib3==1.22             # via requests, requests-unixsocket, urllib3
> ws4py==0.5.1              # via pylxd
> EOF

Note the urllib3==1.22.

root@df4a2f10d39a:/# virtualenv test
New python executable in /test/bin/python
Installing setuptools, pip, wheel...done.

root@df4a2f10d39a:/# ./test/bin/pip install --progress-bar=off -r requirements-lxd.txt                                                                                                                         
Collecting asn1crypto==0.24.0 (from -r requirements-lxd.txt (line 1))                                                                                                                                          
  Downloading https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)                                        
                                                                                                                                                                                                               
Collecting certifi==2018.4.16 (from -r requirements-lxd.txt (line 2))                                                                                                                                          
  Downloading https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl (150kB)                                        
                                                                                                                                                                                                               
Collecting cffi==1.11.5 (from -r requirements-lxd.txt (line 3))                                                                                                                                                
  Downloading https://files.pythonhosted.org/packages/14/dd/3e7a1e1280e7d767bd3fa15791759c91ec19058ebe31217fe66f3e9a8c49/cffi-1.11.5-cp27-cp27mu-manylinux1_x86_64.whl (407kB)                                 
                                                                                                                                                                                                               
Collecting chardet==3.0.4 (from -r requirements-lxd.txt (line 4))                                                                                                                                              
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)                                            
                                                                                                                                                                                                               
Collecting cryptography==2.2.2 (from -r requirements-lxd.txt (line 5))                                                                                                                                         
  Downloading https://files.pythonhosted.org/packages/dd/c2/3a5bfefb25690725824ade71e6b65449f0a9f4b29702cce10560f786ebf6/cryptography-2.2.2-cp27-cp27mu-manylinux1_x86_64.whl (2.2MB)                          
                                                                                                                                                                                                               
Collecting enum34==1.1.6 (from -r requirements-lxd.txt (line 6))                                                                                                                                               
  Downloading https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl                                                         
Collecting idna==2.6 (from -r requirements-lxd.txt (line 7))                                                                                                                                                   
  Downloading https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl (56kB)                                                  
                                                                                                                                                                                                               
Collecting ipaddress==1.0.22 (from -r requirements-lxd.txt (line 8))                                                                                                                                           
  Downloading https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl                                                 
Collecting pbr==4.0.4 (from -r requirements-lxd.txt (line 9))                                                                                                                                                  
  Downloading https://files.pythonhosted.org/packages/b3/5d/c196041ffdf3e34ba206db6d61d1f893a75e1f3435699ade9bd65e089a3d/pbr-4.0.4-py2.py3-none-any.whl (98kB)                                                 

Collecting pycparser==2.18 (from -r requirements-lxd.txt (line 10))
  Downloading https://files.pythonhosted.org/packages/8c/2d/aad7f16146f4197a11f8e91fb81df177adcc2073d36a17b1491fd09df6ed/pycparser-2.18.tar.gz (245kB)

Collecting pylxd==2.2.6 (from -r requirements-lxd.txt (line 11))
  Downloading https://files.pythonhosted.org/packages/a0/5e/af099af60d089b28df6b550d34c1e807ce4e3906d257744f55573c4d3cbb/pylxd-2.2.6.tar.gz (71kB)

    Complete output from command python setup.py egg_info:

    Installed /tmp/pip-install-Knq2xX/pylxd/.eggs/requests_unixsocket-0.1.5-py2.7.egg
    Searching for requests!=2.8.0,>=2.5.2
    Reading https://pypi.org/simple/requests/
    Downloading https://files.pythonhosted.org/packages/49/df/50aa1999ab9bde74656c2919d9c0c085fd2b3775fd3eca826012bef76d8c/requests-2.18.4-py2.py3-none-any.whl#sha256=6a1b267aa90cac58ac3a765d067950e7dbbf75b1da07e895d1f594193a40a38b
    Best match: requests 2.18.4
    Processing requests-2.18.4-py2.py3-none-any.whl
    Installing requests-2.18.4-py2.py3-none-any.whl to /tmp/pip-install-Knq2xX/pylxd/.eggs
    writing requirements to /tmp/pip-install-Knq2xX/pylxd/.eggs/requests-2.18.4-py2.7.egg/EGG-INFO/requires.txt

    Installed /tmp/pip-install-Knq2xX/pylxd/.eggs/requests-2.18.4-py2.7.egg
    Searching for pbr>=1.8
    Reading https://pypi.org/simple/pbr/
    Downloading https://files.pythonhosted.org/packages/b3/5d/c196041ffdf3e34ba206db6d61d1f893a75e1f3435699ade9bd65e089a3d/pbr-4.0.4-py2.py3-none-any.whl#sha256=3747c6f017f2dc099986c325239661948f9f5176f6880d9fdef164cb664cd665
    Best match: pbr 4.0.4
    Processing pbr-4.0.4-py2.py3-none-any.whl
    Installing pbr-4.0.4-py2.py3-none-any.whl to /tmp/pip-install-Knq2xX/pylxd/.eggs

    Installed /tmp/pip-install-Knq2xX/pylxd/.eggs/pbr-4.0.4-py2.7.egg
    Searching for urllib3>=1.8
    Reading https://pypi.org/simple/urllib3/
    Downloading https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl#sha256=b5725a0bd4ba422ab0e66e89e030c806576753ea3ee08554382c14e685d117b5
    Best match: urllib3 1.23
    Processing urllib3-1.23-py2.py3-none-any.whl
    Installing urllib3-1.23-py2.py3-none-any.whl to /tmp/pip-install-Knq2xX/pylxd/.eggs
    writing requirements to /tmp/pip-install-Knq2xX/pylxd/.eggs/urllib3-1.23-py2.7.egg/EGG-INFO/requires.txt

    Installed /tmp/pip-install-Knq2xX/pylxd/.eggs/urllib3-1.23-py2.7.egg
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-Knq2xX/pylxd/setup.py", line 34, in <module>
        pbr=True)
      File "/test/lib/python2.7/site-packages/setuptools/__init__.py", line 128, in setup
        _install_setup_requires(attrs)
      File "/test/lib/python2.7/site-packages/setuptools/__init__.py", line 123, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/test/lib/python2.7/site-packages/setuptools/dist.py", line 514, in fetch_build_eggs
        replace_conflicting=True,
      File "/test/lib/python2.7/site-packages/pkg_resources/__init__.py", line 779, in resolve
        raise VersionConflict(dist, req).with_context(dependent_req)
    pkg_resources.ContextualVersionConflict: (urllib3 1.23 (/tmp/pip-install-Knq2xX/pylxd/.eggs/urllib3-1.23-py2.7.egg), Requirement.parse('urllib3<1.23,>=1.21.1'), set(['requests']))

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-Knq2xX/pylxd/
ajkavanagh commented

Thanks very much for working through this one.

pdecat commented

You're welcome :)
Any idea if/when a 2.2.7 release can be expected to resolve this issue?

pdecat commented

An alternative to an expedited 2.2.7 release may be to build and publish eggs for 2.2.6 including urlllib3 1.22:

# virtualenv .venv27
# ./.venv27/bin/pip install urllib3==1.22
# ./.venv27/bin/python2 setup.py bdist_egg

# python3 -m venv .venv
# ./.venv/bin/pip install urllib==1.22
# ./.venv/bin/python3 setup.py bdist_egg

Note: I'm talking eggs because pylxd's current setup.py does not support wheels.

pdecat commented

Hmm, supporting wheels is just a matter of installing the wheel package in the venv:

# python3 -m venv .venv
# ./.venv/bin/pip install urllib==1.22
# ./.venv/bin/pip install wheel
# ./.venv/bin/python3 setup.py bdist_wheel
ajkavanagh commented

@pdecat I'll push out 2.2.7 tomorrow - I had to do some testing against various older distros and all the lxds!

  • 👍1
pdecat commented

Great, thanks!

ajkavanagh commented

@pdecat Just to let you know that 2.2.7 has been released with the relevant patches in it. Thanks again.

  • 👍1
pdecat commented

Thanks! Issue resolved for us.