`pip wheel -r ...` does not pass `config_settings`
belm0 opened this issue · 20 comments
Description
pip wheel
correctly passes config_settings
when given a requirement specifier directly, but not when given a requirements file.
Works:
pip wheel --config-settings "setup-args=-Dblas=blas" scipy==1.10.1
Doesn't work:
echo 'scipy==1.10.1' > r.txt
pip wheel --config-settings "setup-args=-Dblas=blas" -r r.txt
Expected behavior
config_settings
are passed when using pip wheel
with a requirements file
pip version
23.2.1 (also head and older versions)
Python version
3.8
OS
Linux
How to Reproduce
- in an environment with
gfortran
intentionally missing, try to build scipy withpip wheel -r ...
and settingsetup-args
- in the error output (due to missing
gfortran
), confirm whethersetup-args
propagated to themeson
command line
-r
case (broken)
pip install --upgrade pip wheel
echo 'scipy==1.10.1' > r.txt
pip wheel --config-settings "setup-args=-Dblas=blas" -r r.txt
direct specifier case
pip install --upgrade pip wheel
pip wheel --config-settings "setup-args=-Dblas=blas" scipy==1.10.1
Output
-r
case (broken)
-Dblas
is not passed to meson build
+ meson setup --prefix=/usr /tmp/pip-wheel-6j8l4ith/scipy_206aaca2ef1b4369946502a91692a4b9 /tmp/pip-wheel-6j8l4ith/scipy_206aaca2ef1b4369946502a91692a4b9/.mesonpy-oc75antm/build --native-file=/tmp/pip-wheel-6j8l4ith/scipy_206aaca2ef1b4369946502a91692a4b9/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2
direct specifier case
+ meson setup --prefix=/usr /tmp/pip-wheel-jf4yjx_6/scipy_5f743564b93f4f4eaadba75a455c21b7 /tmp/pip-wheel-jf4yjx_6/scipy_5f743564b93f4f4eaadba75a455c21b7/.mesonpy-l963lj1w/build --native-file=/tmp/pip-wheel-jf4yjx_6/scipy_5f743564b93f4f4eaadba75a455c21b7/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2 -Dblas=blas
Code of Conduct
- I agree to follow the PSF Code of Conduct.
I just discovered this problem too. In my case, I'm using --config-settings editable_mode=compat
in order for pylance to find my editable installations.
And it's not trivial to just convert my reqs into command line args. For example, I'd have to work out a new mechanism for environment variable interpolation and/or deal with changes in req sort order.
This seems to be a regression, because it was working for me last year.
I wonder if the culprit is pyproject-hooks and/or the pyproject.toml
transition.
I've confirmed the discrepancy of -r
vs. direct specifier this far:
- in both cases,
ConfiguredBuildBackendHookCaller.prepare_metadata_for_build_wheel()
is called withconfig_settings=None
prepare_metadata_for_build_wheel()
ignoresconfig_settings
anyway, and defers toself.config_holder.config_settings
- for
-r
case,self.config_holder.config_settings
isNone
- for direct specifier,
self.config_holder.config_settings
reflects the command line config_settings (e.g.{'setup-args': '-Dblas=blas'}
)
aside: it's odd that ConfiguredBuildBackendHookCaller.prepare_metadata_for_build_wheel()
ignores the config_settings
arg
pip/src/pip/_internal/utils/misc.py
Lines 757 to 768 in 496b268
... tracing higher up, in bad case install_req_from_line()
is called with config_settings=None
here are backtraces for bad (passing config_settings=None
) and good cases:
bad (-r
)
Traceback (most recent call last):
File "/.../pip/src/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
status = run_func(*args)
File "/.../pip/src/pip/_internal/cli/req_command.py", line 245, in wrapper
return func(self, options, args)
File "/.../pip/src/pip/_internal/commands/wheel.py", line 120, in run
reqs = self.get_requirements(args, options, finder, session)
File "/.../pip/src/pip/_internal/cli/req_command.py", line 436, in get_requirements
req_to_add = install_req_from_parsed_requirement(
File "/.../pip/src/pip/_internal/req/constructors.py", line 500, in install_req_from_parsed_requirement
req = install_req_from_line(
File "/.../pip/src/pip/_internal/req/constructors.py", line 422, in install_req_from_line
assert False, f'{config_settings=}'
AssertionError: config_settings=None
good
Traceback (most recent call last):
File "/.../pip/src/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
status = run_func(*args)
File "/.../pip/src/pip/_internal/cli/req_command.py", line 245, in wrapper
return func(self, options, args)
File "/.../pip/src/pip/_internal/commands/wheel.py", line 120, in run
reqs = self.get_requirements(args, options, finder, session)
File "/.../pip/src/pip/_internal/cli/req_command.py", line 411, in get_requirements
req_to_add = install_req_from_line(
File "/.../pip/src/pip/_internal/req/constructors.py", line 422, in install_req_from_line
assert False, f'{config_settings=}'
AssertionError: config_settings={'setup-args': '-Dblas=blas'}
in CLI get_requirements()
:
install_req_from_parsed_requirement()
case:
config_settings=parsed_req.options.get("config_settings") if parsed_req.options else None
install_req_from_line()
case:
config_settings=getattr(options, "config_settings", None)
So the code is clearly only expecting config_settings from the requirements entries, and ignoring the command line.
I suppose the settings should be merged, with command line taking precedence...
pip/src/pip/_internal/cli/req_command.py
Lines 431 to 445 in 496b268
I suppose the settings should be merged, with command line taking precedence...
Apparently, when #11634 added per-requirement config_settings, the PR initially tried to merge with the command line settings, but that was removed before the final version.
It appears intentional: #11915 (comment) and reinforced: #11941
My use case for setting config_settings
on the command line, and expecting it to propagate to dependency builds, is valid:
- using
pip wheel -r ...
to build all dependencies of an application to target a less-popular Python implementation (Pyston) - need to control the blas library selected by
scipy
viaconfig_settings
I'll investigate per-requirement config_settings for my use case. But at the least, it may be good for pip
to issue a warning that the command-line config settings are being ignored. @sbidoul @pfmoore
Configuration settings provided via
--config-settings
command line options (or the
equivalent environment variables or configuration file entries) are passed to the build
of requirements explicitly provided as pip command line arguments. They are not passed
to the build of dependencies, or to the build of requirements provided in requirement
files.
I'll investigate per-requirement config_settings for my use case.
It's not practical, because pip-compile
doesn't appear to propagate settings from .in
to .txt
.
$ echo 'scipy==1.10.1 --config-settings="setup-args=-Dblas=blas"' > r2.in
$ pip-compile -r r2.in
numpy==1.24.4
# via scipy
scipy==1.10.1
# via -r r2.in
filed jazzband/pip-tools#2000
I feel that config settings should not be merged (perferrably with a warning), but if config settings are only provided from one source that should be respected instead of being “cancelled”.
So as noted in #12310 (comment) the behaviour is intentional and was changed to resolve an inconsistency with passing config settings to the build of dependencies.
That does not mean it cannot be changed again. I think this would require a detailed specification of the desired behaviour, though. For instance I'm not sure what build backends should do with config settings they don't understand - is that defined in a standard? How should per-requirements config settings behave when there are "global" config settings? Should per-requirements config settings propagate to their dependencies or not?
I'll note that my use case is, as far as I can tell, due to some kind of disagreement between pylance
and pip
and I feel kinda "caught in the middle". Ultimately, I believe I shouldn't need to pass any cli arg for vscode to be able to find my editable installations, but I'm willing do so if pip/pylance say that's what I need to do.
@qci-amos Yes, it's a disagreement between two tools, if you want to look at it that way. The tools in question are setuptools
and pylance
. The problem is that setuptools implements editable installs by default in a way that pylance can't detect. The compat
mode uses a different implementation that is recognised by pylance.
I'd suggest bringing this up with setuptools. They may have a way of setting a global (or per-project) configuration which chooses a different default implementation. Or they may have another way of working around this - I'd be surprised if you are the only person affected by this.
I know setuptools and the typing community have had discussions about this issue - the technical problem behind it is currently unresolved (it needs people to work on, and agree, a standard way of publishing the data that type checkers need from an editable install) as far as I know, but I don't know if it's something they are actively working on, or if there are acceptable workarounds for now (if there are, they would presumably help you as well).
Ok, thank you. Digging around in here I found many potentially related issues so I just picked one to cross-link.
I'd be surprised if you are the only person affected by this.
Yea, I agree! I and my users are not doing anything complicated as far as the build system itself is concerned. All I'm trying to do is install multiple repos in editable mode and not get orange squiggles all over my vscode tabs! My guess is that many devs out there assume it's their bug somehow but give up trying to figure it out and decide instead to just live with the squiggles.
Digging around in here I found many potentially related issues so I just picked one to cross-link.
The specific one you want is pypa/setuptools#3518
Thank you! One suggestion made there that might be relevant here is to define an environment variable as an alternative to a cli arg. That might be easier to get working with -r
and I could see this being useful beyond only the PEP-660 use case.
All of pip's command line options can be specified via environment variables - see the documentation
Ah, ok, it looks like I can get this to work:
export PIP_CONFIG_SETTINGS="editable_mode=compat"
pip install -e .
and this is a win for me because usually I don't install from a requirements file (and the cli arg is difficult for me in my day-to-day ad hoc work). However, this doesn't seem to see the env var:
export PIP_CONFIG_SETTINGS="editable_mode=compat"
echo "-e ." > reqs.txt
pip install -r reqs.txt
However, this doesn't seem to see the env var
Yes, that is because pip processes PIP_CONFIG_SETTINGS identically as the --config-settings
CLI option.
And these are applied to requirements passed as arguments only and do not currently propagate to requirements passed inside -r
requirement files.
Note that with #12494, you should be able to add --config-settings
to -e
lines in requirement files. Would that help in your use case?
For what it's worth, the env var solution I'd been using before, described above #12310 (comment), seems to have stopped working with pip 24, but I haven't had time to debug this.