Build failures on O(S)R(F) buildfarm: "no attribute 'legacy'"

tl;dr: before I set USE_SYSTEM_PACKAGES FALSE and increase the size of my package by 34 MB, do you recognise the below described error?

Long version: not asking you to fix anything (yet ;) ), just curious whether you've ran into this yourself perhaps (which could save me quite some time).

Builds of a released ROS package (haros_catkin) which depends on catkin_virtualenv have been failing for a long time on Kinetic and Melodic on all architectures except amd64 for some reason (I don't even know for how long already, as the mailhost for build.ros.org is on a spamlist, and I've not received any emails from it since Sept 2019).

Example error from a failed build:

Installing pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
  ERROR: Command errored out with exit status 1:
   command: /tmp/binarydeb/ros-melodic-haros-catkin-0.1.1/obj-aarch64-linux-gnu/venv/bin/python /tmp/binarydeb/ros-melodic-haros-catkin-0.1.1/obj-aarch64-linux-gnu/venv/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpabxgc0
       cwd: /tmp/pip-install-yQ6DgO/lazy-object-proxy
  Complete output (10 lines):
  Traceback (most recent call last):
    File "/tmp/binarydeb/ros-melodic-haros-catkin-0.1.1/obj-aarch64-linux-gnu/venv/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 257, in <module>
      main()
    File "/tmp/binarydeb/ros-melodic-haros-catkin-0.1.1/obj-aarch64-linux-gnu/venv/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 240, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/tmp/binarydeb/ros-melodic-haros-catkin-0.1.1/obj-aarch64-linux-gnu/venv/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 85, in get_requires_for_build_wheel
      backend = _build_backend()
    File "/tmp/binarydeb/ros-melodic-haros-catkin-0.1.1/obj-aarch64-linux-gnu/venv/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 76, in _build_backend
      obj = getattr(obj, path_part)
  AttributeError: 'module' object has no attribute '__legacy__'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /tmp/binarydeb/ros-melodic-haros-catkin-0.1.1/obj-aarch64-linux-gnu/venv/bin/python /tmp/binarydeb/ros-melodic-haros-catkin-0.1.1/obj-aarch64-linux-gnu/venv/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpabxgc0 Check the logs for full command output.

This repeats a few times (because of retries).

It fails on arm64, armhf and i386, but not on amd64. This makes it difficult to diagnose, as I don't have access to any of those platforms (or at least: not easily). Tests with amd64 succeed.

Searching for the errors seems to suggest it has something to do with setuptools being old and --use-system-packages, where this could somehow lead to the system-provided setuptools being found before the one in the venv, causing pip failures (fi: pypa/pip#6164 (comment)), or because of running pip commands with the CWD being one which contains a setup.py (pypa/pip#6164 (comment)).

Disabling --use-system-packages is supported by catkin_virtualenv, but will increase the size of the package significantly (34MB+) and is not guaranteed to work (iiuc the linked issue).

Just did a source /path/to/catkin_ws/build/haros_catkin/venv/bin/activate and then pip list | grep setuptools. This returns setuptools 20.7.0 for me (on a local system, I don't have access to the workspace the buildfarm uses).

The buildfarm seems to run /tmp/binarydeb/ros-kinetic-haros-catkin-0.1.1/obj-x86_64-linux-gnu/venv/bin/python /tmp/binarydeb/ros-kinetic-haros-catkin-0.1.1/obj-x86_64-linux-gnu/venv/bin/pip install -qq -U pip==19.3.1. "The internet" seems to suggest that pip >= 19 does not play nice with such old versions of setuptools.

The next command the buildfarm runs is /tmp/binarydeb/ros-kinetic-haros-catkin-0.1.1/obj-x86_64-linux-gnu/venv/bin/python /tmp/binarydeb/ros-kinetic-haros-catkin-0.1.1/obj-x86_64-linux-gnu/venv/bin/pip install -qq setuptools<45.

Could this be reversed and should setuptools be updated/installed first and then the new pip version? virtualenv --python /usr/bin/python2 --system-site-packages --no-setuptools ./venv suggests there should be no setuptools yet in the venv when it's created.

(if it wasn't clear: I'm not into Python too much)

I'm not into Python too much

This is voodoo to me as well. I've never used this package on ARM, and I'm never surprised to see issues, especially related to various pip and setuptools versions.

Could this be reversed and should setuptools be updated/installed first and then the new pip version?

That certainly sounds reasonable, if you find it fixes the issue. I also don't have direct access to those platforms for testing - QEMU might help there. I'm happy with whatever change helps you move forward as long as the tests pass :)

Please keep in mind that 0.6 works a bit differently (https://github.com/locusrobotics/catkin_virtualenv/blob/master/catkin_virtualenv/src/catkin_virtualenv/venv.py#L79)

Seems I can reproduce this with a prerelease test for Kinetic+Xenial on i386.

Unfortunately 0.6 fails with a similar error message :(

I'll try to figure out whether the order of things matters.

Edit: setting USE_SYSTEM_PACKAGES FALSE makes things work again. But has the side-effect of increasing package size by 30MB+. From what I understand from the issues I found it could be that either setuptools is not installed at all when venv/pip tries to use it, or because the only requirement passed to pip is <45, any version of setuptools will satisfy that requirement, leading to very old versions getting used.

Edit 2: changing setuptools<45 to setuptools==44.0.0 as a test (with USE_SYSTEM_PACKAGES back to its default (ie: FALSE)) and no other changes also seems to work.

Output from pip when run here:

Collecting pip==20.1
  Downloading pip-20.1-py2.py3-none-any.whl (1.5 MB)
Collecting pip-tools==5.1.2
  Downloading pip_tools-5.1.2-py2.py3-none-any.whl (43 kB)
Collecting setuptools==44.0.0
  Downloading setuptools-44.0.0-py2.py3-none-any.whl (583 kB)
Collecting click>=7
  Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)
Requirement already satisfied: six in /usr/lib/python2.7/dist-packages (from pip-tools==5.1.2) (1.10.0)
Installing collected packages: pip, click, pip-tools, setuptools
  Attempting uninstall: pip
    Found existing installation: pip 20.2.2
    Uninstalling pip-20.2.2:
      Successfully uninstalled pip-20.2.2
  Attempting uninstall: setuptools
    Found existing installation: setuptools 20.7.0
    Not uninstalling setuptools at /usr/lib/python2.7/dist-packages, outside environment /tmp/ws/build_isolated/haros_catkin/venv
    Can't uninstall 'setuptools'. No files were found to uninstall.
Successfully installed click-7.1.2 pip-20.1 pip-tools-5.1.2 setuptools-44.0.0

So it seems indeed the case that the 20.7.0 version of setuptools on i386 Xenial is just too old for pip==20.1 which then causes the failure.

Now to see which minimum version of setuptools will work in this case, to get a setuptools>=X,<45 requirement.

Now to see which minimum version of setuptools will work in this case, to get a setuptools>=X,<45 requirement.

What about settling on a 44.x version of setuptools? That would still mean <45, but would automatically install updates to the 44.x series of setuptools (not sure those are still released).

Ok. Changing:

catkin_virtualenv/catkin_virtualenv/src/catkin_virtualenv/venv.py

Line 79 in 8cda360

preinstall += ['setuptools<45']

To:

preinstall += ['setuptools>=44,<45']

works for me in a prerelease run with i386 Kinetic on Xenial.

This installs:

Collecting pip==20.1
  Downloading pip-20.1-py2.py3-none-any.whl (1.5 MB)
Collecting pip-tools==5.1.2
  Downloading pip_tools-5.1.2-py2.py3-none-any.whl (43 kB)
Collecting setuptools<45,>=44
  Downloading setuptools-44.1.1-py2.py3-none-any.whl (583 kB)
Collecting click>=7
  Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)
Requirement already satisfied: six in /usr/lib/python2.7/dist-packages (from pip-tools==5.1.2) (1.10.0)
Installing collected packages: pip, click, pip-tools, setuptools
  Attempting uninstall: pip
    Found existing installation: pip 20.2.2
    Uninstalling pip-20.2.2:
      Successfully uninstalled pip-20.2.2
  Attempting uninstall: setuptools
    Found existing installation: setuptools 20.7.0
    Not uninstalling setuptools at /usr/lib/python2.7/dist-packages, outside environment /tmp/ws/build_isolated/haros_catkin/venv
    Can't uninstall 'setuptools'. No files were found to uninstall.
Successfully installed click-7.1.2 pip-20.1 pip-tools-5.1.2 setuptools-44.1.1

I'll submit a PR.

See #71.

Afaict tests also still pass like this.

Re-opening as this is still an issue unfortunately.

Problem is I can't reproduce it locally. I've tried using the aarch64 Docker images to run a Debian Stretch + Melodic prerelease, but those succeed ..

Edit: oh, seems I can't re-open my own issue?

Hm, not sure what happened, but now they are green again.

I'll keep an eye on the builds.

It fails on arm64, armhf and i386, but not on amd64. This makes it difficult to diagnose, as I don't have access to any of those platforms (or at least: not easily).

Final comment (for future readers perhaps also): multi-arch support via QEMU and binfmt in the end allowed me to run prerelease tests for arm64 and armhf architectures.

They took "forever" (due to emulation), but it was better than nothing.

On some versions of Ubuntu you need to fiddle a little bit with the binfmt configuration. I took inspiration from docker/for-linux#56 and the linked computermouth/qemu-static-conf (copying only the files I needed though, and to /etc/binfmt.d instead of creating a new directory under /lib).