Cannot install easybuild-easyconfigs using pip if setuptools>61
ktaletsk opened this issue · 6 comments
Hi, I'm using easybuild in my Docker image, which uses miniconda environment for Python. Recently, after some updates I could not build my image anymore because pip install easybuild
has failed with a large number of errors similar to package init file 'easybuild/__init__.py' not found (or not a regular file)
.
Full log
19:19:53 Step 7/7 : RUN pip install --ignore-installed easybuild==4.5.1
19:19:53 ---> Running in c8da42b3c8a2
19:19:54 Collecting easybuild==4.5.1
19:19:54 Downloading easybuild-4.5.1.tar.gz (10 kB)
19:19:54 Preparing metadata (setup.py): started
19:19:55 Preparing metadata (setup.py): finished with status 'done'
19:19:55 Collecting easybuild-framework==4.5.1
19:19:55 Downloading easybuild-framework-4.5.1.tar.gz (2.0 MB)
19:19:55 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 40.4 MB/s eta 0:00:00
19:19:55 Preparing metadata (setup.py): started
19:19:57 Preparing metadata (setup.py): finished with status 'done'
19:19:57 Collecting easybuild-easyblocks==4.5.1
19:19:57 Downloading easybuild-easyblocks-4.5.1.tar.gz (512 kB)
19:19:57 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 512.9/512.9 KB 50.6 MB/s eta 0:00:00
19:19:57 Preparing metadata (setup.py): started
19:19:58 Preparing metadata (setup.py): finished with status 'done'
19:19:58 Collecting easybuild-easyconfigs==4.5.1
19:19:58 Downloading easybuild-easyconfigs-4.5.1.tar.gz (7.1 MB)
19:19:58 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.1/7.1 MB 87.0 MB/s eta 0:00:00
19:20:03 Preparing metadata (setup.py): started
19:20:05 Preparing metadata (setup.py): finished with status 'done'
19:20:05 Building wheels for collected packages: easybuild, easybuild-easyblocks, easybuild-easyconfigs, easybuild-framework
19:20:05 Building wheel for easybuild (setup.py): started
19:20:06 Building wheel for easybuild (setup.py): finished with status 'done'
19:20:06 Created wheel for easybuild: filename=easybuild-4.5.1-py3-none-any.whl size=10225 sha256=90c543a8f52f5748e7f39ce70c82c13ba856cd8598c47d967640644d04626eaa
19:20:06 Stored in directory: /home/jovyan/.cache/pip/wheels/e5/4d/03/71de5ed71b80ef1a238622c21c1d654cee06f80b32ba89681c
19:20:06 Building wheel for easybuild-easyblocks (setup.py): started
19:20:08 Building wheel for easybuild-easyblocks (setup.py): finished with status 'done'
19:20:08 Created wheel for easybuild-easyblocks: filename=easybuild_easyblocks-4.5.1-py3-none-any.whl size=784121 sha256=beecf56724cea8c7b4ed8cd061f440ab7d4d40cef3bcabab2c98af424cfba377
19:20:08 Stored in directory: /home/jovyan/.cache/pip/wheels/33/d1/f1/333e64b6bcbfa8c7735fd1065d429b1af23f87d45db763ef1c
19:20:08 Building wheel for easybuild-easyconfigs (setup.py): started
19:20:09 Building wheel for easybuild-easyconfigs (setup.py): finished with status 'error'
19:20:09 �[91m error: subprocess-exited-with-error
19:20:09
19:20:09 × python setup.py bdist_wheel did not run successfully.
19:20:09 │ exit code: 1
19:20:09 ╰─> [35 lines of output]
19:20:09 Installing version 4.5.1 (required versions: API >= 4, easyblocks >= 4.5)
19:20:09 running bdist_wheel
19:20:09 running build
19:20:09 running build_py
19:20:09 package init file 'easybuild/__init__.py' not found (or not a regular file)
19:20:09 package init file 'easybuild/easyconfigs/__init__.py' not found (or not a regular file)
19:20:09 package init file 'easybuild/easyconfigs/x/__init__.py' not found (or not a regular file)
19:20:09 package init file 'easybuild/easyconfigs/i/__init__.py' not found (or not a regular file)
19:20:09 package init file 'easybuild/easyconfigs/e/__init__.py' not found (or not a regular file)
19:20:09 package init file 'easybuild/easyconfigs/r/__init__.py' not found (or not a regular file)
19:20:09 package init file 'easybuild/easyconfigs/a/__init__.py' not found (or not a regular file)
19:20:09 package init file 'easybuild/easyconfigs/w/__init__.py' not found (or not a regular file)
19:20:09 package init file 'easybuild/easyconfigs/c/__init__.py' not found (or not a regular file)
19:20:09 package init file 'easybuild/easyconfigs/j/__init__.py' not found (or not a regular file)
19:20:09 package init file 'easybuild/easyconfigs/y/__init__.py' not found (or not a regular file)
19:20:09 package init file 'easybuild/easyconfigs/t/__init__.py' not found (or not a regular file)
19:20:09 package init file 'easybuild/easyconfigs/m/__init__.py' not found (or not a regular file)
19:20:09 package init file 'easybuild/easyconfigs/q/__init__.py' not found (or not a regular file)
19:20:09 package init file 'easybuild/easyconfigs/s/__init__.py' not found (or not a regular file)
19:20:09 package init file 'easybuild/easyconfigs/z/__init__.py' not found (or not a regular file)
19:20:09 package init file 'easybuild/easyconfigs/g/__init__.py' not found (or not a regular file)
19:20:09 package init file 'easybuild/easyconfigs/v/__init__.py' not found (or not a regular file)
19:20:09 package init file 'easybuild/easyconfigs/o/__init__.py' not found (or not a regular file)
19:20:09 package init file 'easybuild/easyconfigs/d/__init__.py' not found (or not a regular file)
19:20:09 package init file 'easybuild/easyconfigs/u/__init__.py' not found (or not a regular file)
19:20:09 package init file 'easybuild/easyconfigs/k/__init__.py' not found (or not a regular file)
19:20:09 package init file 'easybuild/easyconfigs/b/__init__.py' not found (or not a regular file)
19:20:09 package init file 'easybuild/easyconfigs/l/__init__.py' not found (or not a regular file)
19:20:09 package init file 'easybuild/easyconfigs/p/__init__.py' not found (or not a regular file)
19:20:09 creating build
19:20:09 creating build/lib
19:20:09 creating build/lib/easybuild
19:20:09 creating build/lib/easybuild/easyconfigs
19:20:09 creating build/lib/easybuild/easyconfigs/p
19:20:09 error: can't copy 'easybuild/easyconfigs/p/plotly.py': doesn't exist or not a regular file
19:20:09 [end of output]
19:20:09
19:20:09 note: This error originates from a subprocess, and is likely not a problem with pip.
19:20:09 �[0m�[91m ERROR: Failed building wheel for easybuild-easyconfigs
19:20:09 �[0m Running setup.py clean for easybuild-easyconfigs
19:20:10 Building wheel for easybuild-framework (setup.py): started
19:20:13 Building wheel for easybuild-framework (setup.py): finished with status 'done'
19:20:13 Created wheel for easybuild-framework: filename=easybuild_framework-4.5.1-py3-none-any.whl size=3595163 sha256=23a751e990d25a9acebe225168d382f08eecaee45a3556d753c21fd30d5510bf
19:20:13 Stored in directory: /home/jovyan/.cache/pip/wheels/b4/f8/e1/4b87f276053032e2ddc1459b9270850d010f0875972bde7340
19:20:13 Successfully built easybuild easybuild-easyblocks easybuild-framework
19:20:13 Failed to build easybuild-easyconfigs
19:20:14 Installing collected packages: easybuild-framework, easybuild-easyconfigs, easybuild-easyblocks, easybuild
19:20:15 Running setup.py install for easybuild-easyconfigs: started
19:20:16 Running setup.py install for easybuild-easyconfigs: finished with status 'error'
19:20:16 �[91m error: subprocess-exited-with-error
19:20:16
19:20:16 × Running setup.py install for easybuild-easyconfigs did not run successfully.
19:20:16 │ exit code: 1
19:20:16 ╰─> [37 lines of output]
19:20:16 Installing version 4.5.1 (required versions: API >= 4, easyblocks >= 4.5)
19:20:16 running install
19:20:16 /opt/conda/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
19:20:16 warnings.warn(
19:20:16 running build
19:20:16 running build_py
19:20:16 package init file 'easybuild/__init__.py' not found (or not a regular file)
19:20:16 package init file 'easybuild/easyconfigs/__init__.py' not found (or not a regular file)
19:20:16 package init file 'easybuild/easyconfigs/x/__init__.py' not found (or not a regular file)
19:20:16 package init file 'easybuild/easyconfigs/i/__init__.py' not found (or not a regular file)
19:20:16 package init file 'easybuild/easyconfigs/e/__init__.py' not found (or not a regular file)
19:20:16 package init file 'easybuild/easyconfigs/r/__init__.py' not found (or not a regular file)
19:20:16 package init file 'easybuild/easyconfigs/a/__init__.py' not found (or not a regular file)
19:20:16 package init file 'easybuild/easyconfigs/w/__init__.py' not found (or not a regular file)
19:20:16 package init file 'easybuild/easyconfigs/c/__init__.py' not found (or not a regular file)
19:20:16 package init file 'easybuild/easyconfigs/j/__init__.py' not found (or not a regular file)
19:20:16 package init file 'easybuild/easyconfigs/y/__init__.py' not found (or not a regular file)
19:20:16 package init file 'easybuild/easyconfigs/t/__init__.py' not found (or not a regular file)
19:20:16 package init file 'easybuild/easyconfigs/m/__init__.py' not found (or not a regular file)
19:20:16 package init file 'easybuild/easyconfigs/q/__init__.py' not found (or not a regular file)
19:20:16 package init file 'easybuild/easyconfigs/s/__init__.py' not found (or not a regular file)
19:20:16 package init file 'easybuild/easyconfigs/z/__init__.py' not found (or not a regular file)
19:20:16 package init file 'easybuild/easyconfigs/g/__init__.py' not found (or not a regular file)
19:20:16 package init file 'easybuild/easyconfigs/v/__init__.py' not found (or not a regular file)
19:20:16 package init file 'easybuild/easyconfigs/o/__init__.py' not found (or not a regular file)
19:20:16 package init file 'easybuild/easyconfigs/d/__init__.py' not found (or not a regular file)
19:20:16 package init file 'easybuild/easyconfigs/u/__init__.py' not found (or not a regular file)
19:20:16 package init file 'easybuild/easyconfigs/k/__init__.py' not found (or not a regular file)
19:20:16 package init file 'easybuild/easyconfigs/b/__init__.py' not found (or not a regular file)
19:20:16 package init file 'easybuild/easyconfigs/l/__init__.py' not found (or not a regular file)
19:20:16 package init file 'easybuild/easyconfigs/p/__init__.py' not found (or not a regular file)
19:20:16 creating build
19:20:16 creating build/lib
19:20:16 creating build/lib/easybuild
19:20:16 creating build/lib/easybuild/easyconfigs
19:20:16 creating build/lib/easybuild/easyconfigs/p
19:20:16 error: can't copy 'easybuild/easyconfigs/p/plotly.py': doesn't exist or not a regular file
19:20:16 [end of output]
19:20:16
19:20:16 note: This error originates from a subprocess, and is likely not a problem with pip.
19:20:16 �[0m�[91merror: legacy-install-failure
19:20:16
19:20:16 × Encountered error while trying to install package.
19:20:16 ╰─> easybuild-easyconfigs
19:20:16
19:20:16 note: This is an issue with the package mentioned above, not pip.
19:20:16 hint: See above for output from the failure.
19:20:17 �[0mThe command '/bin/sh -c pip install --ignore-installed easybuild==4.5.1' returned a non-zero code: 1
For a context, here are versions that I have in my container
version | |
---|---|
python | 3.9.12 |
conda | 4.10.3 |
pip | 22.0.4 |
setuptools | 61.2.0 |
I noticed on Stackoverflow someone recently had the same issue recently https://stackoverflow.com/questions/71648527/how-to-solve-easybuild-installation-problems
The issue seem to be isolated specifically in building a Python wheel for easybuild-easyconfigs
package. I tried to isolate any issues related to my container by running popular public containers. Below are the results of my testing:
- Python Docker image,
docker run --rm -it python pip install easybuild-easyconfigs
. Success.
version | |
---|---|
python | 3.10.4 |
conda | N/A |
pip | 22.0.4 |
setuptools | 58.1.0 |
Full log
Collecting easybuild-easyconfigs
Downloading easybuild-easyconfigs-4.5.3.tar.gz (7.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.3/7.3 MB 15.9 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Building wheels for collected packages: easybuild-easyconfigs
Building wheel for easybuild-easyconfigs (setup.py) ... done
Created wheel for easybuild-easyconfigs: filename=easybuild_easyconfigs-4.5.3-py3-none-any.whl size=22498617 sha256=c11bf548aea702685b75c08847304b62ea4702501d3e755cc7419dad60de945e
Stored in directory: /root/.cache/pip/wheels/38/81/59/0001c055d5ca749b81d1f0149dce8cd71d803e370ee4b3be4f
Successfully built easybuild-easyconfigs
Installing collected packages: easybuild-easyconfigs
Successfully installed easybuild-easyconfigs-4.5.3
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
- Jupyter Minimal Notebook Docker image from March 21,
docker run --rm -it jupyter/minimal-notebook:2022-03-21 pip install easybuild-easyconfigs==4.5.3
. Success.
version | |
---|---|
python | 3.9.10 |
conda | 4.11.0 |
pip | 22.0.3 |
setuptools | 60.9.3 |
Full log
Collecting easybuild-easyconfigs==4.5.3
Downloading easybuild-easyconfigs-4.5.3.tar.gz (7.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.3/7.3 MB 16.8 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Building wheels for collected packages: easybuild-easyconfigs
Building wheel for easybuild-easyconfigs (setup.py) ... done
Created wheel for easybuild-easyconfigs: filename=easybuild_easyconfigs-4.5.3-py3-none-any.whl size=22498617 sha256=32a9aa13ca84779bd2a59f3c4ebd00995626c0a4b4fcaf830dd15b27d9dd5759
Stored in directory: /home/jovyan/.cache/pip/wheels/71/e6/f0/d353f5e1b8aff637cadf674d069ebf1e0a34e653f3ccded3ee
Successfully built easybuild-easyconfigs
Installing collected packages: easybuild-easyconfigs
Successfully installed easybuild-easyconfigs-4.5.3
- Jupyter Minimal Notebook Docker image from March 28,
docker run --rm -it jupyter/minimal-notebook:2022-03-28 pip install easybuild-easyconfigs==4.5.3
. Failure.
version | |
---|---|
python | 3.9.10 |
conda | 4.12.0 |
pip | 22.0.4 |
setuptools | 61.1.1 |
Full log
Collecting easybuild-easyconfigs==4.5.3
Downloading easybuild-easyconfigs-4.5.3.tar.gz (7.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.3/7.3 MB 13.0 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Building wheels for collected packages: easybuild-easyconfigs
Building wheel for easybuild-easyconfigs (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [20 lines of output]
Installing version 4.5.3 (required versions: API >= 4, easyblocks >= 4.5)
running bdist_wheel
running build
running build_py
package init file 'easybuild/__init__.py' not found (or not a regular file)
package init file 'easybuild/easyconfigs/__init__.py' not found (or not a regular file)
package init file 'easybuild/easyconfigs/v/__init__.py' not found (or not a regular file)
package init file 'easybuild/easyconfigs/a/__init__.py' not found (or not a regular file)
package init file 'easybuild/easyconfigs/g/__init__.py' not found (or not a regular file)
package init file 'easybuild/easyconfigs/k/__init__.py' not found (or not a regular file)
package init file 'easybuild/easyconfigs/z/__init__.py' not found (or not a regular file)
package init file 'easybuild/easyconfigs/i/__init__.py' not found (or not a regular file)
package init file 'easybuild/easyconfigs/__archive__/__init__.py' not found (or not a regular file)
package init file 'easybuild/easyconfigs/p/__init__.py' not found (or not a regular file)
creating build
creating build/lib
creating build/lib/easybuild
creating build/lib/easybuild/easyconfigs
creating build/lib/easybuild/easyconfigs/p
error: can't copy 'easybuild/easyconfigs/p/path.py': doesn't exist or not a regular file
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for easybuild-easyconfigs
Running setup.py clean for easybuild-easyconfigs
Failed to build easybuild-easyconfigs
Installing collected packages: easybuild-easyconfigs
Running setup.py install for easybuild-easyconfigs ... error
error: subprocess-exited-with-error
× Running setup.py install for easybuild-easyconfigs did not run successfully.
│ exit code: 1
╰─> [22 lines of output]
Installing version 4.5.3 (required versions: API >= 4, easyblocks >= 4.5)
running install
/opt/conda/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_py
package init file 'easybuild/__init__.py' not found (or not a regular file)
package init file 'easybuild/easyconfigs/__init__.py' not found (or not a regular file)
package init file 'easybuild/easyconfigs/v/__init__.py' not found (or not a regular file)
package init file 'easybuild/easyconfigs/a/__init__.py' not found (or not a regular file)
package init file 'easybuild/easyconfigs/g/__init__.py' not found (or not a regular file)
package init file 'easybuild/easyconfigs/k/__init__.py' not found (or not a regular file)
package init file 'easybuild/easyconfigs/z/__init__.py' not found (or not a regular file)
package init file 'easybuild/easyconfigs/i/__init__.py' not found (or not a regular file)
package init file 'easybuild/easyconfigs/__archive__/__init__.py' not found (or not a regular file)
package init file 'easybuild/easyconfigs/p/__init__.py' not found (or not a regular file)
creating build
creating build/lib
creating build/lib/easybuild
creating build/lib/easybuild/easyconfigs
creating build/lib/easybuild/easyconfigs/p
error: can't copy 'easybuild/easyconfigs/p/path.py': doesn't exist or not a regular file
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure
× Encountered error while trying to install package.
╰─> easybuild-easyconfigs
note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
I am confused what's going on. There seem to be no clear correlation of the issue with versions of Python, Conda or pip.
UPD: Added setuptools
versions to the scenarios and it seems like the version 61 is the one causing issues
Workaround for anyone affected -- build a wheel easybuild-easyconfigs
in an environment where it works and copy it to your main environment. If using Docker, you can utilize a multistage build to do that
# Add this pre-build stage
----------
FROM python:3.9
RUN pip wheel easybuild-easyconfigs==4.5.3
----------
# Your normal Dockerfile
FROM ...
# Add this COPY command
----------
COPY --from=0 /easybuild* .
----------
RUN pip install easybuild_easyconfigs-4.5.3-py3-none-any.whl easybuild==4.5.3
After applying this changes, build proceeds without issues
Any relation to a similar issue that was reported at easybuilders/easybuild-framework#3984 ?
It seems like some kind of breaking change was introduced in a recent version of setuptools
or pip
that we're hitting?
@ktaletsk Can you update the issue description with the setuptools
version that was used in each case?
@boegel Added setuptools versions. Looks like easybuild-easyconfigs
is incompatible with setuptools
above version 61. As a check, downgrading setuptools
to <61, and trying install again, works:
> docker run --rm -it jupyter/minimal-notebook:2022-03-28 bash
$ pip install easybuild-easyconfigs==4.5.3
# Same failure as above
# Downgrade setuptools
$ pip install --upgrade setuptools==60.9.3
# Try installing again
$ pip install easybuild-easyconfigs==4.5.3
Collecting easybuild-easyconfigs==4.5.3
Using cached easybuild-easyconfigs-4.5.3.tar.gz (7.3 MB)
Preparing metadata (setup.py) ... done
Building wheels for collected packages: easybuild-easyconfigs
Building wheel for easybuild-easyconfigs (setup.py) ... done
Created wheel for easybuild-easyconfigs: filename=easybuild_easyconfigs-4.5.3-py3-none-any.whl size=22498617 sha256=97de381848c130dc53ad275c43c0b4a6e18c12200388072af10f9cc8410e791d
Stored in directory: /home/jovyan/.cache/pip/wheels/71/e6/f0/d353f5e1b8aff637cadf674d069ebf1e0a34e653f3ccded3ee
Successfully built easybuild-easyconfigs
Installing collected packages: easybuild-easyconfigs
Successfully installed easybuild-easyconfigs-4.5.3
Thanks a lot, that's helpful.
It looks like we're being hit by a breaking change in setuptools 61.0, see https://github.com/pypa/setuptools/blob/main/CHANGES.rst#v6100, probably related to pypa/setuptools#2894...
The easyconfigs package is a special case, since we're only packaging up data files (easyconfigs + patch files, basically), rather than Python code.
It seems like we have some homework to do to ensure compatibility with future setuptools versions...
There's a trivial fix for this, see easybuilders/easybuild-easyconfigs#15206, we'll make sure that's included for the next EasyBuild release (but we also want to try setting up a CI workflow that catches problems like this)