Doesn't work with Cython3
Opened this issue · 4 comments
Thank you for the tutorial and the repo. I've verified that the project does work with cython 0.x series but fails on cython3.
$ /tmp/example-cython-poetry-pypi
❯ poetry build -vvv
Using virtualenv: /Users/lmwangi/Library/Caches/pypoetry/virtualenvs/poetry-pypi-example-wSzIWKA2-py3.9
Building poetry-pypi-example (0.0.1)
- Building sdist
- Adding: /private/tmp/example-cython-poetry-pypi/poetry_pypi_example/main.cpython-39-darwin.so
- Adding: README.rst
- Adding: pyproject.toml
- Built poetry-pypi-example-0.0.1.tar.gz
- Building wheel
- Executing build script: build.py
[1/1] Cythonizing poetry_pypi_example/main.py
Traceback (most recent call last):
File "/private/tmp/example-cython-poetry-pypi/build.py", line 80, in <module>
build()
File "/private/tmp/example-cython-poetry-pypi/build.py", line 30, in build
build_ext_cmd.copy_extensions_to_source()
File "/Users/lmwangi/Library/Caches/pypoetry/virtualenvs/poetry-pypi-example-wSzIWKA2-py3.9/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 103, in __getattr__
raise AttributeError(attr)
AttributeError: copy_extensions_to_source
On drilling down, it looks like the import for build_ext in cython3 does not have ** copy_extensions_to_source**. To be precise in line 14, build_ext is None when it is imported from setuptools. Please see below:
Python 3.9.13 (main, May 24 2022, 21:28:31)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.6.0 -- An enhanced Interactive Python. Type '?' for help.
# Injected an IPython shell just before the setuptools import in Cython/Distutils/build_ext.py
In [1]: _build_ext_module
# Should return the build_ext module. Hmm, we can't get the build_ext module. Let's check the parents.
In [2]: sys.modules.get('setuptools.command.build_ext')
# We can get the setuptools module
In [3]: sys.modules.get('setuptools')
Out[3]: <module 'setuptools' from '/Users/lmwangi/Library/Caches/pypoetry/virtualenvs/wallet-api-ZqjMZXwH-py3.9/lib/python3.9/site-packages/setuptools/__init__.py'>
# We can get the setuptools.command module
In [5]: sys.modules.get('setuptools.command')
Out[5]: <module 'setuptools.command' from '/Users/lmwangi/Library/Caches/pypoetry/virtualenvs/wallet-api-ZqjMZXwH-py3.9/lib/python3.9/site-packages/setuptools/command/__init__.py'>
# On the filesystem, the module does exist.
In [6]: !ls /Users/lmwangi/Library/Caches/pypoetry/virtualenvs/wallet-api-ZqjMZXwH-py3.9/lib/python3.9/site-packages/setuptools/command
__init__.py alias.py bdist_rpm.py build_ext.py develop.py easy_install.py install.py install_lib.py 'launcher manifest.xml' register.py saveopts.py setopt.py upload.py
__pycache__ bdist_egg.py build_clib.py build_py.py dist_info.py egg_info.py install_egg_info.py install_scripts.py py36compat.py rotate.py sdist.py test.py upload_docs.py
# Hmm, import does work
In [8]: import setuptools.command.build_ext
# Now a get works....
In [9]: sys.modules.get('setuptools.command.build_ext')
Out[9]: <module 'setuptools.command.build_ext' from '/Users/lmwangi/Library/Caches/pypoetry/virtualenvs/wallet-api-ZqjMZXwH-py3.9/lib/python3.9/site-packages/setuptools/command/build_ext.py'>Digging further, it looks like this is the [breaking commit](https://github.com/cython/cython/commit/c6f5c5ddcc021099febae7cb1194fedea01fd056).
< # Always inherit from the "build_ext" in distutils since setuptools already imports
< # it from Cython if available, and does the proper distutils fallback otherwise.
< # https://github.com/pypa/setuptools/blob/9f1822ee910df3df930a98ab99f66d18bb70659b/setuptools/command/build_ext.py#L16
<
< # setuptools imports Cython's "build_ext", so make sure we go first.
< _build_ext_module = sys.modules.get('setuptools.command.build_ext')
< if _build_ext_module is None:
< import distutils.command.build_ext as _build_ext_module
<
< # setuptools remembers the original distutils "build_ext" as "_du_build_ext"
< _build_ext = getattr(_build_ext_module, '_du_build_ext', None)
< if _build_ext is None:
< _build_ext = getattr(_build_ext_module, 'build_ext', None)
< if _build_ext is None:
---
> if 'setuptools' in sys.modules:
> try:
> from setuptools.command.build_ext import build_ext as _build_ext
> except ImportError:
> # We may be in the process of importing setuptools, which tries
> # to import this.
> from distutils.command.build_ext import build_ext as _build_extIf we revert the import in Cython/Distutils/build_ext.py with the one in cython 0.x, the build now works
❯ head -17 /Users/lmwangi/Library/Caches/pypoetry/virtualenvs/poetry-pypi-example-wSzIWKA2-py3.9/lib/python3.9/site-packages/Cython/Distutils/build_ext.py
import sys
import os
import sys
if 'setuptools' in sys.modules:
try:
from setuptools.command.build_ext import build_ext as _build_ext
except ImportError:
# We may be in the process of importing setuptools, which tries
# to import this.
from distutils.command.build_ext import build_ext as _build_ext
else:
from distutils.command.build_ext import build_ext as _build_ext
❯ poetry run Cython -V
Cython version 3.0.0a10
❯ poetry build
Building poetry-pypi-example (0.0.1)
- Building sdist
- Built poetry-pypi-example-0.0.1.tar.gz
- Building wheel
[1/1] Cythonizing poetry_pypi_example/main.py
- Built poetry_pypi_example-0.0.1-cp39-cp39-macosx_12_0_x86_64.whl@labeneator Hello! Did you figure out how to fix this error and use that script with latest version of Cython and Poetry?
@labeneator Hello! Did you figure out how to fix this error and use that script with latest version of Cython and Poetry?
With the small patch above, it worked. However, I had to abandon the cython route because pydantic <> cython does not work. Cython maintainers were unwilling to revert the patch as per: https://groups.google.com/g/cython-users/c/CQ2ieXACtzk/m/guDy3kFZBAAJ
@labeneator Thanks for your answer! I already fixed this tomorrow by editing build.py a bit:
...
def build(setup_kwargs: Dict[str, Any]) -> Dict[str, Any]:
# Generate stub files for type hints
generate_stub_files()
# Collect and cythonize all files
extension_modules = cythonize_helper(get_extension_modules())
# Use Setuptools to collect files
distribution = Distribution(
{
"ext_modules": extension_modules,
"cmdclass": {
"build_ext": cython_build_ext,
},
}
)
# Grab the build_ext command
build_ext_cmd = distribution.get_command_obj("build_ext")
build_ext_cmd.ensure_finalized()
# Set the value to 1 for "inplace", with the goal to build extensions
# in build directory, and then copy all files back to the source dir
# (under the hood, "copy_extensions_to_source" will be called after
# building the extensions). This is done so Poetry grabs the files
# during the next step in its build.
build_ext_cmd.inplace = 1
build_ext_cmd.run()
return setup_kwargs
...
if __name__ == '__main__':
build({})And here are my build settings in pyproject.toml:
[tool.poetry]
...
# Only include cythonized files in final build
include = ["mylib/**/*.so", "mylib/**/*.pyd"] # ignored in VCS, so we need to be explicit
exclude = ["mylib/**/*.py"]
...
[tool.poetry.build]
generate-setup-file = false
script = "build.py"
...Now I'm using the latest version of Poetry and the latest version of Cython (3.0.0a11).
And as a result, I get a protected package without source code, only with .pyd extensions, and I also added auto-generation of .pyi files so that hints and types remain when using the library.
I heard that pydantic is good, but simple typing and sometimes dataclasses are enough for me. The main thing is that the source code is protected.
And I also recommend not converting Path to the str(Path) string, but using the .as_posix method, because backslashes on Windows can cause problems. Because of this, the cythonization process did not start for me at all.
So you need to replace this
# Convert path to module name
module_path = str(module_path).replace("/", ".")to this
# Convert path to module name
module_path = module_path.as_posix().replace("/", ".")and in other places too