python-cffi/cffi

`cffi>=1.16.0` failing to install on `mingw`

abravalheri opened this issue · 2 comments

abravalheri commented

Recently I was changing something unrelated in setuptools-rust and I noticed that pip install cffi started to fail in mingw. When I tried to cap the version with cffi<1.16 everything installed well.

Which lead me to investigate what is causing this.
I created this repo just to use github CI (since I don't have access to a mingw environment to run the tests manually).

The reproducer (encoded in the Github workflow of the repository mentioned above) is relatively simple:

  • Given: a mingw environment setup using msys2 and with the following packages installed:
    • mingw-w64-{i686/x86_64}-python ⇒ This should give you Python 3.11.6
    • mingw-w64-{i686/x86_64}-python-pip
    • mingw-w64-{i686/x86_64}-openssl
    • mingw-w64-{i686/x86_64}-toolchain
      (The choice between i686 and x86_64 depends on the mingw architecture you are running)
  1. Install the latest versions of pip, setuptools, wheel: 
    python -m pip install -U pip setuptools wheel
python -m pip list
# Package    Version
# ---------- -------
# distlib    0.3.7
# pip        23.2.1
# setuptools 68.2.2
# wheel      0.41.2
  2. Attempt to install the latest version of cffi 
    python -m pip -v install -U "cffi>=1.16.0"
# ...
#     File "D:/a/_temp/msys64/tmp/pip-build-env-l9qomo_f/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in run_setup
#       exec(code, locals())
#     File "<string>", line 126, in <module>
#     File "<string>", line 105, in uses_msvc
#     File "D:/a/_temp/msys64/tmp/pip-build-env-l9qomo_f/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/config.py", line 223, in try_compile
# ...
# distutils.errors.DistutilsPlatformError: --plat-name must be one of ('win32', 'win-amd64', 'win-arm32', 'win-arm64')
# ...
    ... and observe the error (triggered by https://github.com/python-cffi/cffi/blob/v1.16.0/setup.py#L105 - https://github.com/python-cffi/cffi/blob/v1.16.0/setup.py#L126).

As show in the Github workflow there are a few interesting things:

  • The error does not happen for cffi<1.16
    (even if I install setuptools and export SETUPTOOLS_USE_DISTUTILS=local, which I expect to replace any imports from distutils with setuptools._distutils even if setuptools is not imported -- including during the build of cffi).
  • The error does not happen if I pass --no-build-isolation to pip

I can see that mingw==1.16.0 added pyproject.toml which makes pip treat the installation process differently (instead of running python setup.py ... it will call the PEP 517 hooks), and the error seems to happen when pip calls the PEP 517 hook: setuptools.build_meta.get_requires_for_build_wheel (which is internally translated by setuptools into a patched egg_info call -- it is a lot of patching...).

Something else that I am not understanding (probably unrelated) is why in some cases distutils is still being loaded from setuptools._distutils even when SETUPTOOLS_USE_DISTUTILS=stdlib.

The reason why I opened the issue is to bring this problem to your attention and ask if you have any suggestions, tips that can help to solve this problem.

In principle it seems to be a compatibility issue with the procedure setuptools employs to support backwards compatibility with packages that need setup_requires, but I still couldn't figure out exactly was is causing the problem.

The fact that cffi<1.16 is compatible with setuptools._distutils puzzles me.

Moreover, cffi==1.16 works fine with pip install --no-build-isolation.
So when running the other PEP 517 hooks, distutils/_msvccompiler.py never seems to be used, but for that specific patched egg_info, this problematic code path is triggered. Any idea why is that the case?

abravalheri commented

One thing that might be important to mention is that mingw does not seem to be officially supported in distutils (pypa/distutils#34), but cffi seems to support it.

nitzmahone commented

Yeah, the ongoing tug-of-war around PEP517/518 config_settings mappings and how pip/build/setuptools maps them to old distutils command args and decide which build hooks receive them is causing grief in a lot of places. The setuptools-embedded distutils has been provided and used by default for a very long time via their _distutils_hack .pth shim, so that's probably a red herring. As you noted, CFFI 1.16 is the first release that uses a PEP517 build, and when you're running that with build isolation, pip passes the build args through setuptools' generic PEP517 hooks, which then tries to map those args and fails on unsupported values.

IIRC, pip currently skips calling the get_requires_for_build_wheel PEP517 hook when you disable build isolation, as that also implicitly disables automatic requirements installation (and just uses whatever setuptools/wheel is laying around in your current Python env). Since that setuptools hook's arg translator is the one that's barfing on the unknown platform, skipping it lets the build continue.

I'm just guessing based on my own fights with this same stuff in build and pip trying to modernize ancient custom distutils build config stuff in other projects- I don't have mingw/msys2 stuff laying around either. I'm probably unlikely to add it to CFFI's test matrix at this point, as Windows build/test is already by far the slowest and most expensive (further complicated by libffi requiring a GNU toolchain to build).

I don't think there's anything more CFFI can do here- running with build isolation disabled should continue to work for now. You might want to consider a fix PR to setuptools that allows the mingw/msys2 platform arg through that gate, but that's about all I can think of for now. In light of that, I'm going to close this issue.