Is there a way to programmatically tell if CFFI is using the limited API?

[astro-stan]

From the docs and reading source I gather that CFFI tries to compile C extensions with Py_LIMITED_API defined, but there are exceptions.

Is there a way to tell from python whether the generated code is Py_LIMITED_API compatible?

[arigo]

You can add any C code to the extension module while it is being compiled. For example:

ffibuilder.cdef("""
    #define IS_LIMITED ...
""")

ffibuilder.set_source(..., """
...

#ifdef PYPY_VERSION
# define IS_LIMITED  (-1)   /* doesn't make much sense on pypy */
#elif defined(Py_LIMITED_API)
# define IS_LIMITED  1
#else
# define IS_LIMITED  0
#endif
}
""")

[astro-stan]

Thanks but I want to know this before the compilation has begun (i.e. during the code generation phase).

I am using meson-python as the build backend and want to be able to set the limited api arg.

Does CFFI decide this purely in Python or is there some preprocessor logic embedded in the generated code?

[arigo]

OK. Sorry if I'm still missing your question, but the logic is indeed embedded in the start of the generated code. See the file cffi/_cffi_include.h, the first page of comments, about Py_LIMITED_API. The content of this file gets copied into the generated code.

[arigo]

See also def _set_py_limited_api() in setuptools_ext.py.

[astro-stan]

Ah this is what I was afraid of.

OK, in that case can you summarize the rules for compiling with limited API, please?

Something like "if on windows and py3.7+" - yes, "if on apple silicon" - no. (I am making these up as an example).

Also, how likely is it for the rules to change in the future?

[arigo]

Still unsure what you really need, given that there is both a #define Py_LIMITED_API that activates in some cases in the generated .c file, and also some setuptools-specific hack that takes place before that. The latter has logic that can nowadays be summarized like this: always set the 'py_limited_api' flag to setuptools, unless we are building with a debug version of CPython. The other cases are all about situations that should no longer really occur nowadays (e.g. CPython 3.4, or setuptools version <= 25).

[arigo]

The setuptools-based rules will change in the future if and when setuptools messes up and we need to work around that.

Really, the rules are of the kind "try to set py_limited_api because it's always better, except when we can detect that doing so would somehow create a serious problem at whatever stage".

[astro-stan]

I am just trying to make sure I toggle the limited API option in meson-python, so that it produces correctly named wheels. But from what I understood - there isn't a clear guideline on when that is the case.

So I suppose the solution is to create an ABI3 named wheel and test if it works in all subsequent minor Python versions on the target platform.

Would that be a correct conclusion?

[arigo]

Yes. The generated code should always be fully compatible with the limited API. The cases where we don't activate it are only the cases where it has been found to create trouble. Nowadays you can mostly assume that there shouldn't be trouble because the limited API is well-tested by now---but be sure to test, just in case.

[astro-stan]

Thanks for the confirmation.

I think this may need a separate issue, but just to confirm first:

On a 64-bit Windows, with Python 3.7 and using MinGW, the Py_LIMITED_API should be defined and the extension should be linked against the python3.dll, correct?

Asking as I am using CIBW to build ABI3 wheels (3.7+). The wheels are built (and named) correctly but when I test them against 3.8+ I get:

ImportError while importing test module 'D:\a\project\project\tests\project_test.py'.
  Hint: make sure your test modules/packages have valid Python names.
  Traceback:
  ..\venv-test\lib\site-packages\project\_internals\__init__.py:5: in <module>
      from project._project import ffi as _ffi 
  E   ImportError: DLL load failed while importing _project: The specified module could not be found.

The same setup works correctly on Linux (building with GH actions). ABI3 wheels get built on 3.7 and tested against 3.7+.

Build deps:

[build-system]
requires = [
  "meson-python>=0.16.0",
  'meson>=1.5.0rc2',
  'ninja',
  "cffi>=1.15.1; python_version<'3.8'",
  "cffi>=1.16; python_version>='3.8'",
  "setuptools; python_version>='3.12'",
]

This PR seems relevant: mesonbuild/meson#13171

But the patch is in meson>=1.5.0rc1 and up as far as I can tell.

[arigo]

Sorry, that's beyond me. Maybe someone else will answer here, or you need to ask elsewhere like on StackOverflow.

[astro-stan]

Sorry, that's beyond me. Maybe someone else will answer here, or you need to ask elsewhere like on StackOverflow.

Fixed under mesonbuild/meson#13379

Thanks for your help. Closing this.