[BUG] TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero'
tim-s-ccs opened this issue · 38 comments
setuptools version
setuptools>=71.0.0
Python version
Python 3.12
OS
Ubuntu
Additional environment information
I found this bug when running the test for the project in GitHub actions
Description
When I install the requirements in my Python package project (pip-sync requirements-dev.txt) I get an error within setuptools which says:
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
...
TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero'
This error was caused by using setuptools >= 71.0.0
Not to speculate but it looks like the issue was introduced by this commit in setuptools: 00384a5
And this commit is including new changes from the packaging project and its this commit here where we can see the issue:
pypa/packaging@cc938f9
https://github.com/pypa/packaging/blame/4493dfcd95a893f676a7aa4bd17c547bea676371/src/packaging/utils.py#L58
Expected behavior
I would expect pip-sync requirements-dev.txt to run successfully
How to Reproduce
Clone https://github.com/tim-s-ccs/example-python-package.git
Make sure you are using Python 3.12 and you have pip-tools installed (pip install --upgrade pip wheel pip-tools)
Make sure you are using the correct version of setuptools pip install --upgrade setuptools==71.0.0
Run pip-compile requirements-dev.in
Output
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [42 lines of output]
running egg_info
creating /private/var/folders/kt/n16_msf56_z3kfyb65l37wxhb_m4s_/T/pip-pip-egg-info-3sca_19y/example_python_package.egg-info
writing /private/var/folders/kt/n16_msf56_z3kfyb65l37wxhb_m4s_/T/pip-pip-egg-info-3sca_19y/example_python_package.egg-info/PKG-INFO
writing dependency_links to /private/var/folders/kt/n16_msf56_z3kfyb65l37wxhb_m4s_/T/pip-pip-egg-info-3sca_19y/example_python_package.egg-info/dependency_links.txt
writing top-level names to /private/var/folders/kt/n16_msf56_z3kfyb65l37wxhb_m4s_/T/pip-pip-egg-info-3sca_19y/example_python_package.egg-info/top_level.txt
writing manifest file '/private/var/folders/kt/n16_msf56_z3kfyb65l37wxhb_m4s_/T/pip-pip-egg-info-3sca_19y/example_python_package.egg-info/SOURCES.txt'
reading manifest file '/private/var/folders/kt/n16_msf56_z3kfyb65l37wxhb_m4s_/T/pip-pip-egg-info-3sca_19y/example_python_package.egg-info/SOURCES.txt'
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "/Users/my.name/Code/test/example-python-package/setup.py", line 16, in <module>
setup(
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/setuptools/__init__.py", line 106, in setup
return distutils.core.setup(**attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 184, in setup
return run_commands(dist)
^^^^^^^^^^^^^^^^^^
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
dist.run_commands()
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 970, in run_commands
self.run_command(cmd)
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/setuptools/dist.py", line 974, in run_command
super().run_command(command)
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 989, in run_command
cmd_obj.run()
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 321, in run
self.find_sources()
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 329, in find_sources
mm.run()
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 555, in run
self.prune_file_list()
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 621, in prune_file_list
base_dir = self.distribution.get_fullname()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/setuptools/_core_metadata.py", line 266, in get_fullname
return _distribution_fullname(self.get_name(), self.get_version())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/setuptools/_core_metadata.py", line 284, in _distribution_fullname
canonicalize_version(version, strip_trailing_zero=False),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero'
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
Traceback (most recent call last):
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/pip/_internal/operations/build/metadata_legacy.py", line 64, in generate_metadata
call_subprocess(
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/pip/_internal/utils/subprocess.py", line 209, in call_subprocess
raise error
pip._internal.exceptions.InstallationSubprocessError: python setup.py egg_info exited with 1
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/my.name/.pyenv/versions/3.12.2/bin/pip-compile", line 8, in <module>
sys.exit(cli())
^^^^^
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/piptools/scripts/compile.py", line 470, in cli
results = resolver.resolve(max_rounds=max_rounds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/piptools/resolver.py", line 604, in resolve
is_resolved = self._do_resolve(
^^^^^^^^^^^^^^^^^
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/piptools/resolver.py", line 636, in _do_resolve
resolver.resolve(
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 76, in resolve
collected = self.factory.collect_root_requirements(root_reqs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 538, in collect_root_requirements
reqs = list(
^^^^^
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 494, in _make_requirements_from_install_req
cand = self._make_base_candidate_from_link(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 210, in _make_base_candidate_from_link
self._editable_candidate_cache[link] = EditableCandidate(
^^^^^^^^^^^^^^^^^^
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 328, in __init__
super().__init__(
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 158, in __init__
self.dist = self._prepare()
^^^^^^^^^^^^^^^
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 235, in _prepare
dist = self._prepare_distribution()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 338, in _prepare_distribution
return self._factory.preparer.prepare_editable_requirement(self._ireq)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 698, in prepare_editable_requirement
dist = _get_prepared_distribution(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 72, in _get_prepared_distribution
abstract_dist.prepare_distribution_metadata(
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/pip/_internal/distributions/sdist.py", line 69, in prepare_distribution_metadata
self.req.prepare_metadata()
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/pip/_internal/req/req_install.py", line 580, in prepare_metadata
self.metadata_directory = generate_metadata_legacy(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/my.name/.pyenv/versions/3.12.2/lib/python3.12/site-packages/pip/_internal/operations/build/metadata_legacy.py", line 71, in generate_metadata
raise MetadataGenerationFailed(package_details=details) from error
pip._internal.exceptions.MetadataGenerationFailed: metadata generation failedIssue is visible when you use packaging<22.0
Could reproduce the same issue. Fixed by downgrading to 70.x
Also it can be fixed by update packaging to 22.0 or higher
This behavior is by design. Similar to the issue reported in #4478, the new Setuptools 71 will prefer installed dependencies over the vendored ones. You'll want to either uninstall the older (incompatible) dependencies from the environment or install these dependencies. You may be tempted to install setuptools[core], but please avoid doing that as core is an internal implementation detail.
This behavior is by design. Similar to the issue reported in #4478, the new Setuptools 71 will prefer installed dependencies over the vendored ones. You'll want to either uninstall the older (incompatible) dependencies from the environment or install these dependencies. You may be tempted to install
setuptools[core], but please avoid doing that ascoreis an internal implementation detail.
Just to confirm, we're seeing this happen in this environment:
python -m pip install --upgrade pip
python -m pip install --upgrade setuptools wheel twine check-wheel-contentsIs that correct? It sounds like you're saying the issue is old dependencies existing in the environment, but we should have the most current packages here.
Thanks @jaraco this was helpful. In my project where I was having the issue because on older version of packaging was a dependancy (not compatible with setuptools v71.0.0 or maybe earlier). Updating that dependancy did indeed fix the issue.
It's a bit annoying that the behaviour of setuptools changed which messed up my tests (which gets the latest version of setuptools when setting up) but that is very much my fault and not your teams.
Again, thanks for your comment as it helped my to understand the issue better and what the cause was.
For anyone who has a situation like me where my tests just started failing I'd check to make sure your dependancies and the dependancies of setuptools are compatible.
Thanks @jaraco this was helpful. In my project where I was having the issue because on older version of
packagingwas a dependancy (not compatible with setuptools v71.0.0 or maybe earlier). Updating that dependancy did indeed fix the issue.It's a bit annoying that the behaviour of setuptools changed which messed up my tests (which gets the latest version of setuptools when setting up) but that is very much my fault and not your teams.
Again, thanks for your comment as it helped my to understand the issue better and what the cause was.
For anyone who has a situation like me where my tests just started failing I'd check to make sure your dependancies and the dependancies of setuptools are compatible.
Thanks for posting your solution. I'm trying to reason through how we're running into the same issue. We're not specifying packaging anywhere, so I would assume we would get the most current version. And when I install locally, I get the same version of packaging you specify in your PR. Perhaps the CI image I'm using already has packaging installed, but with an older version? I'll investigate and post back if I find something. But it would definitely be a sharp corner if pip install --upgrade setuptools no longer works on ubuntu-latest.
Perhaps the CI image I'm using already has packaging installed, but with an older version?
Yes, probably.
Thanks for posting your solution. I'm trying to reason through how we're running into the same issue. We're not specifying
packaginganywhere, so I would assume we would get the most current version. And when I install locally, I get the same version ofpackagingyou specify in your PR. Perhaps the CI image I'm using already haspackaginginstalled, but with an older version? I'll investigate and post back if I find something. But it would definitely be a sharp corner ifpip install --upgrade setuptoolsno longer works onubuntu-latest.
Agreed, it is a bit of a sharp corner, and we made the major release as a signal of this shift in expectations. The issue is that while you're holding it correctly and pip install -U setuptools should theoretically upgrade all the required dependencies, it cannot currently due to the bootstrapping problem. I'm actively working on solving that issue, but in the meantime, one needs to pip install -U setuptools[core] to prepare an environment safely.
You may be tempted to install
setuptools[core], but please avoid doing that ascoreis an internal implementation detail.
I think I may have been too cautious about this advice. We specifically want to avoid packages themselves making this declaration in build-requires. I believe it's safe to use in systems integration tooling.
@jaraco Thanks for taking the time to respond. This advice is super useful:
You may be tempted to install
setuptools[core], but please avoid doing that ascoreis an internal implementation detail.I think I may have been too cautious about this advice. We specifically want to avoid packages themselves making this declaration in
build-requires. I believe it's safe to use in systems integration tooling.
The issue that we wound up running into is that older versions of our package actually take a runtime dependency on packaging. Upon verifying the package (by installing it), the version of packaging was getting clobbered. This was only causing an issue for the sdist, and not the wheel.
Thanks for all the information. This change has broken our CI this morning too. Much as I appreciate the workarounds given here, I would have preferred this behaviour to not have gone mainstream until pypa/packaging-problems#342 is resolved, so I don't have to go back and undo these workarounds in the future.
It may be worth referencing this issue in the quickstart documentation since following the documentation may now not work in many existing environments? https://setuptools.pypa.io/en/latest/userguide/quickstart.html#installation
That's a really good point. There's a large scope of users affected by this issue but only a small number of system integrators affected by the bootstrapping problem. Perhaps we shouldn't be imposing this extra burden on the larger group of users and should instead push it to integrators (require them, for example, to patch out the declared dependencies if there's no solution to the bootstrap problem).
I agree - updating the installation guide to capture the current state makes sense.
Another option might be to give the vendored dependencies precedence, but doing so would not reliably solve the issue, as one could python -c "import packaging; import setuptools", which would cause the older packaging to be loaded even thought setuptools would prefer the newer vendored version... and I don't want to go down the road of some intrusive hacks like were employed for distutils (unloading previously-imported packages so they're reloaded).
Oof. This brought all of our CI to a halt, which is the last thing I needed to deal with during a massive migration from gitlab to github.
Can confirm installing 6.1.14 is a temporary solution.
python -m pip install --force-reinstall -v "platformio==6.1.14"
This behavior is by design. Similar to the issue reported in #4478, the new Setuptools 71 will prefer installed dependencies over the vendored ones. You'll want to either uninstall the older (incompatible) dependencies from the environment or install [these dependencies]
installing thedependencies in the link works ! please make new build requierment file inseatd of setup tools py and include all dependencies along with setup tools like below: packaging>=24
ordered-set>=3.1.1
more_itertools>=8.8
jaraco.text>=3.7
importlib_resources>=5.10.2
importlib_metadata>=6
tomli>=2.0.1
wheel>=0.43.0
pkg_resources
platformdirs >= 2.6.2
setuptools~=70.0
running this file with apt update and pip install makes the packages run without error
We are using AWS MWAA (Airflow) in our stack and it comes with very tight constraints, including packaging ,
I will try to lock setuptools on 70.x.x but maybe there is another way around?
packaging==21.3
# via
# -c constraints.txt
# apache-airflow
# connexion
# huggingface-hub
# limits
# marshmallow
# redshift-connector
# sqlalchemy-redshift
# tableauserverclient
```
We are using AWS MWAA (Airflow) in our stack and it comes with very tight constraints, including
packaging, I will try to lock setuptools on 70.x.x but maybe there is another way around?
Another option is not to install setuptools into that environment. Setuptools is aiming not to be installed in runtime environments and to be used primarily as a build backend. The setuptools distribution does also provide the pkg_resources package, but that too is deprecated. Please consider examining your use of setuptools to see if it can be removed from your dependencies when building these MWAA stack applications. I suspect it's not a direct dependency but a transitive one, in which case please report the issue to the project that's requiring it.
I agree - updating the installation guide to capture the current state makes sense.
This was done in #4507 (quickstart), so I'm closing this issue as there's not much more setuptools can do.
Hey @jaraco I am using pipenv and running with these versions bellow
pip==24.1.2, setuptools==70.2.0, wheel==0.43.0
But still getting the same error
TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero'
Per the guidance, please make sure that the setuptools[core] dependencies are installed. In particular, packaging>=24.
hi @jaraco , In my case the setuptools. is getting installed as a dependency to snowflake-connector-python==2.9.0 and it is pulling latest version setuptools 72.1.0 which i guess causing the issue.I am afraid force install of specific version of setuptools which may cause issues. This is in MWAA and while creating a virtual environment it is giving the error. Any inputs/guidance is greatly appreciated.
**Executing cmd: /usr/bin/python3.10 -m virtualenv /tmp/cosmos-venvxjhcuezk --python=python3****
**File "/usr/local/airflow/.local/lib/python3.10/site-packages/setuptools/_core_metadata.py", line 284, in _distribution_fullname
INFO - canonicalize_version(version, strip_trailing_zero=False),
INFO - TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero'**
**subprocess.CalledProcessError: Command '['/usr/bin/python3.10', '-m', 'virtualenv', '/tmp/cosmos-venvxjhcuezk', '--python=python3']' returned non-zero exit status 1.**
I sympathize. It seems that MWAA in particular has very tight constraints (see #4483 (comment)) on packaging.
I'm not familiar enough with that environment to provide an adequate workaround.
@doctornkz-intelas Did you find a workaround? If not, either of you should feel free to file a new ticket describing the specific problem with MWAA and (ideally) providing some repro instructions or describing the interfaces that MWAA gives you to affect the environment. Even better might be to file an issue with MWAA and loop me in there, where I can support the issue in the context of that product. My suspicion is that the long-term solution is to ensure that packaging>=24 is present in the environment, but there are lots of other possible avenues we can pursue (pinning setuptools, building in isolated environments, etc).
Hugs, @vemulagopal. Yes, we found the way, @jaraco. But it might not be applicable to others.
Introduction: MWAA has very aggressive limits. See details. It is because MWAA is preparing runtime for DAGs and also for itself. You can't upgrade a random library without the risk of breaking MWAA and ruining your day. The best approach for the community to render dependencies with pip-tools compile to get the setup as strict as possible, eg:
pip-compile --constraint=constraints.txt --output-file=requirements.txt requirements.in . After compiling the tool is masking pip and setuptools as not-safe dependencies:
...
# The following packages are considered to be unsafe in a requirements file:
# pip
# setuptools
- Our first very dirty approach was to get risk and add setuptools at the end of the file after rendering. (At that moment I created the comment). It works but the solution is ugly.
- After discussion with devs we found another way - we are installing
setuptools~70.x.x on the clean CI/CD runner and during rendering dependencies insidevenvwe are forcefully using system libs. Like that:
python3 -m venv my_project --system-site-packages. But again it's very tricky and not perfect, but at leastsetuptoolsis placed outside ofrequirements.txt.
If someone knows a more sophisticated approach, please, share - it will help many Airflow users.
Thank you @doctornkz-intelas @jaraco for your inputs and guidance, We are planning to upgrade MWAA to 2.8.1 and I have tested the virtual environment in airflow 2.8.1 the issue is gone as the packaging package came with upgraded version in constraints file.
@jaraco I don't find the recommended solution to work.
I'm attempting to install, inside a Conda environment:
pip install -e . --no-warn-script-location
It looks to me like the version for all the dependencies you listed is acceptable, running conda list (list below).
Most packages were already an appropriate version. I only had to specify the following 3 in the environment.yml to explicitly install them:
dependencies:
- python=3.9
- ...
- ordered-set>=3.1.1
- more-itertools>=8.8
- jaraco.text>=3.7
(my_env) [username@dev-server-1 my_project]$ conda list
# packages in environment at /home/username/.conda/envs/my_project:
#
# Name Version Build Channel
...
importlib-metadata 8.4.0 pyha770c72_0 conda-forge
importlib_metadata 8.4.0 hd8ed1ab_0 conda-forge
importlib_resources 6.4.4 pyhd8ed1ab_0 conda-forge
...
jaraco.context 5.3.0 pyhd8ed1ab_1 conda-forge
jaraco.functools 4.0.0 pyhd8ed1ab_0 conda-forge
jaraco.text 3.12.0 py39hf3d152e_0 conda-forge
...
more-itertools 10.4.0 pyhd8ed1ab_0 conda-forge
...
ordered-set 4.1.0 pyhd8ed1ab_0 conda-forge
packaging 24.1 pyhd8ed1ab_0 conda-forge
...
python 3.9.19 h0755675_0_cpython conda-forge
...
setuptools 72.2.0 pyhd8ed1ab_0 conda-forge
...
toml 0.10.2 pyhd8ed1ab_0 conda-forge
tomli 2.0.1 pyhd8ed1ab_0 conda-forge
...
wheel 0.44.0 pyhd8ed1ab_0 conda-forge
...
But, I end up still with the exact same error for canonicalize_version().
Note: The core version dependencies you link to appear to have a typo. "more_itertools" -> "more-itertools"
Otherwise, it doesn't find that dependency in any channels.
"more_itertools" -> "more-itertools"
You're right. I thought I remembered the project name being more_itertools, but I checked the history, and it's been called more-itertools since the 1.0 release, so it was just my misunderstanding (complicated by the fact that normalization rules allow me to use the wrong name everywhere without consequence). There are too many places to fix this issue, so I'll just plan to adjust my understanding and tweak the name when I encounter it.
I agree, it looks like you have compatible dependencies. packaging is the one that matters for this error message, and you have 24.1. But some how, when running the build, you have an older version of packaging.
I suspect what's happening is you have an older version of pip that when it builds the isolated build environment installs some older version of packaging.
You can likely confirm my suspicion by running pip install --no-build-isolation -e . --no-warn-script-location. That should perform the build without build isolation and use the dependencies you have in place.
Assuming that works, that confirms the build isolation is implicated. To see if we can identify where the old packaging is coming from, maybe try: python3.9 -m venv test_env and then testenv/bin/pip list to see if maybe an old packaging gets installed that way (then you can rm -r testenv to cleanup). We should also check what version of pip you have; it's possible an older version of pip somehow pulls in older versions of packaging.
I've been unable to replicate the issue, even using Python 3.9, but I haven't been able to fully replicate your environment (Python patch version, conda environment, etc). Anything you can do to isolate the issue to something I can replicate locally would help me triage more.
Well, for interesting results...
pip version, my_env activated: 24.2 from /home/username/.conda/envs/my_project/lib/python3.9/site-packages/pip (python 3.9)
Running the --no-build-isolation:
(my_env) [username@dev-server-1 my_project]$ pip install --no-build-isolation -e . --no-warn-script-location
Obtaining file:///home/username/projects/my_project
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [39 lines of output]
running egg_info
creating /tmp/pip-pip-egg-info-y_kpr0fn/my_project.egg-info
writing /tmp/pip-pip-egg-info-y_kpr0fn/my_project.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-pip-egg-info-y_kpr0fn/my_project.egg-info/dependency_links.txt
writing entry points to /tmp/pip-pip-egg-info-y_kpr0fn/my_project.egg-info/entry_points.txt
writing requirements to /tmp/pip-pip-egg-info-y_kpr0fn/my_project.egg-info/requires.txt
writing top-level names to /tmp/pip-pip-egg-info-y_kpr0fn/my_project.egg-info/top_level.txt
writing manifest file '/tmp/pip-pip-egg-info-y_kpr0fn/my_project.egg-info/SOURCES.txt'
reading manifest file '/tmp/pip-pip-egg-info-y_kpr0fn/my_project.egg-info/SOURCES.txt'
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "/home/username/projects/my_project/setup.py", line 41, in <module>
setup(
File "/home/username/.conda/envs/my_project/lib/python3.9/site-packages/setuptools/__init__.py", line 108, in setup
return distutils.core.setup(**attrs)
File "/home/username/.conda/envs/my_project/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 184, in setup
return run_commands(dist)
File "/home/username/.conda/envs/my_project/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
dist.run_commands()
File "/home/username/.conda/envs/my_project/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 964, in run_commands
self.run_command(cmd)
File "/home/username/.conda/envs/my_project/lib/python3.9/site-packages/setuptools/dist.py", line 945, in run_command
super().run_command(command)
File "/home/username/.conda/envs/my_project/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 983, in run_command
cmd_obj.run()
File "/home/username/.conda/envs/my_project/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 310, in run
self.find_sources()
File "/home/username/.conda/envs/my_project/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 318, in find_sources
mm.run()
File "/home/username/.conda/envs/my_project/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 544, in run
self.prune_file_list()
File "/home/username/.conda/envs/my_project/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 610, in prune_file_list
base_dir = self.distribution.get_fullname()
File "/home/username/.conda/envs/my_project/lib/python3.9/site-packages/setuptools/_core_metadata.py", line 266, in get_fullname
return _distribution_fullname(self.get_name(), self.get_version())
File "/home/username/.conda/envs/my_project/lib/python3.9/site-packages/setuptools/_core_metadata.py", line 284, in _distribution_fullname
canonicalize_version(version, strip_trailing_zero=False),
TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero'
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
I ran test_env/bin/pip list anyway, and it yielded:
Package Version
---------- -------
pip 23.0.1
setuptools 58.1.0
[notice] A new release of pip is available: 23.0.1 -> 24.2
...
That certainly pulled in an older version of setuptools.
(To note, I ran python3.9 -m venv test_env within the activated Conda environment. Otherwise, python3.9 is not available.
So, environment within an environment. But, I don't know why that wouldn't work -- especially for what you had me run.
But, probably irrelevant, since the main check I ran failed.)
I'm sure there are a lot of possibilities of stuff installed and available in the background, since I'm working on a development server for this.
I think here is probably the answer.
I added the following to setup.py:
import setuptools
import packaging
from importlib.metadata import version
print(f"Version of setuptools: {version('setuptools')} : {setuptools.__file__}")
print(f"Version of packaging: {version('packaging')} : {packaging.__file__}")
import sys
print(sys.path) # To print the set of dirs Python pulls from
import os
print(os.environ['PYTHONPATH'])
And that gave me:
Version of setuptools: 72.2.0 : /home/username/.conda/envs/my_project/lib/python3.9/site-packages/setuptools/__init__.py
Version of packaging: 21.3 : /home/username/.local/lib/python3.9/site-packages/packaging/__init__.py
['', '/apps/base/lib', '/apps/data/lib', '/apps/tools/lib', '/home/username/.conda/envs/my_project/lib/python39.zip', '/home/username/.conda/envs/my_project/lib/python3.9', '/home/username/.conda/envs/my_project/lib/python3.9/lib-dynload', '/home/username/.local/lib/python3.9/site-packages', '/home/username/.conda/envs/my_project/lib/python3.9/site-packages', '/home/username/.conda/envs/my_project/lib/python3.9/site-packages/setuptools/_vendor']
/apps/base/lib:/apps/data/lib:/apps/tools/lib
I think that's all the info I need to remedy this: ~/.local/lib/... comes before .conda/envs/my_project/lib/...
I also tried explicitly adding packaging>=24.0 to the environment.yml, but no change. Ah, the intricacies of Python sourcing.
Also, to note, the build script I'm using to run this definitely specifies the pip installed in the environment. And, with the environment activated, which pip references the environment's pip. So, I don't think it's because pip is getting called from elsewhere.
One way or another, I think I have all the info now to figure out some way to properly import the correct version of packaging.
Thank you very much for looking that over, and providing some feedback, Jason!
Solution: #4483 (comment)
Thank you really much, this solved the issue.
We were running setuptools before 70.x.x.
In our setup we have Databricks which installs a whl to execute our libraries.
Here the config for others:
setuptools~=74.1.1
packaging~=24.1Could reproduce the same issue. Fixed by downgrading to 70.x
You save me!!! all you need to do is upgrade to setuptools==70.0.0. Don't try > 74 or less than 70. None of them can work!!! Packaging I use is 24.1.
For me:
pip3.10 install --upgrade setuptools==70.0.0
You can use:
pip install --upgrade setuptools==70.0.0
Could reproduce the same issue. Fixed by downgrading to 70.x
You save me!!! all you need to do is upgrade to setuptools==70.0.0. Don't try > 74 or less than 70. None of them can work!!! Packaging I use is 24.1.
For me:
pip3.10 install --upgrade setuptools==70.0.0You can use:
pip install --upgrade setuptools==70.0.0
This actually works. Thanks!
Quote reply
Could reproduce the same issue. Fixed by downgrading to 70.x
You save me!!! all you need to do is upgrade to setuptools==70.0.0. Don't try > 74 or less than 70. None of them can work!!! Packaging I use is 24.1.
For me:
pip3.10 install --upgrade setuptools==70.0.0You can use:
pip install --upgrade setuptools==70.0.0
Thanks it worked
Why do I need to downgrade to 70.0.0? This is nuts...
Why do I need to downgrade to 70.0.0? This is nuts...
did you try this?
#4483 (comment)
Downgrading setuptools or adding packaging with a specific version as a dependency are both weird. The docs just say that people need to add setuptools as a build dependency, nothing about packaging. If setuptools depends on packaging, and it is incompatible with older versions of packaging, then "packaging>=MIN_VERSION" should be a required dependency of setuptools, not in a core optional dependency recommended to not be used.
@jaraco how is this behavior "by design"? What I see in the docs does not help.
I was constantly getting the same error on a previously working Jupiter Notebook with python 3.10 no matter what dependencies I was trying to change.
Turns out my conda environment was enforcing broken versions of setuptools and packaging.
Changed the entire conda env to run python 3.12 and the issue went away ✅
@jaraco how is this behavior "by design"?
- Setuptools wishes to use a standards-based canonicalization of the version number.
- The best available standards-based implementation of
canonicalize_versionis in thepackagingmodule. - Setuptools therefore is declaring its dependency on a compatible version of packaging.
- Due to bootstrapping problems in the ecosystem, setuptools cannot declare dependencies on anything that's built by Setuptools or which has dependencies that are built by setuptools.
(aside) Now that I write (4), I realize that packaging uses flit-core, so maybe it's viable to move that dependency (and others that aren't affected by the bootstrapping problem) to required dependencies and not optional.
When I say it's by design, I mean I'm unaware of any other design that meets Setuptools stated goal to be able to follow standard dependency practices and declare dependencies on any packages it requires. This goal is something I've been working on for years.
I recognize and sympathize with the downstream effects of this transition, but I have to say, I'm really appreciating the progress this project has made toward having a simpler and more manageable vendoring strategy and possibly eventually removing the need for vendoring altogether.
I'm open to suggestions, but "just bundle everything all the time" is a regression in the design and a non-starter. What is your suggestion?
Also it can be fixed by update packaging to 22.0 or higher
The related commit in packaging is pypa/packaging@e99b37e if anyone needs a reference (and 22.0 is correct).
Could reproduce the same issue. Fixed by downgrading to 70.x
You save me!!! all you need to do is upgrade to setuptools==70.0.0. Don't try > 74 or less than 70. None of them can work!!! Packaging I use is 24.1.
For me:
pip3.10 install --upgrade setuptools==70.0.0You can use:
pip install --upgrade setuptools==70.0.0
Thanks! It works for me.
WSL2 Utbuntu 20.04, Python 3.8, setuptools==75.x raised the same error when I was building opencv==4.11 (such a pain for it take long time to build), and downgrade to 70.0.0 had no problem.