xmlsec/python-xmlsec

Version 1.3.14 fails to compile

corneauf opened this issue · 15 comments

corneauf commented

While installing xmlsec 1.3.14 we encounter the following error:

#16 21.06       /tmp/pip-install-z5k61bf_/xmlsec/src/keys.c: In function ‘PyXmlSec_KeyFromEngine’:

#16 21.06       /tmp/pip-install-z5k61bf_/xmlsec/src/keys.c:218:65: error: ‘xmlSecKeyDataFormatEngine’ undeclared (first use in this function); did you mean ‘xmlSecKeyDataFormat’?

#16 21.06         218 |         key->handle = xmlSecCryptoAppKeyLoad(engine_and_key_id, xmlSecKeyDataFormatEngine, NULL, xmlSecCryptoAppGetDefaultPwdCallback(), (void*)engine_and_key_id);

#16 21.06             |                                                                 ^~~~~~~~~~~~~~~~~~~~~~~~~

#16 21.06             |                                                                 xmlSecKeyDataFormat

Looking at the project, the xmlSecKeyDataFormatEngine does not seem to be defined anywhere. I've looked the <xmlsec/crypto.h> include and it does not contain the defined struct either.

corneauf commented

From the discussion here #284 it seems some code was copy pasted without the necessary context.

jplrank commented

I'm having the same issue too when installing on a Red Hat 8 system with Python 3.9.12. Version 1.3.13 installs just fine

pip install xmlsec==1.3.14
Collecting xmlsec==1.3.14
  Downloading xmlsec-1.3.14.tar.gz (68 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 68.9/68.9 kB 6.1 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting lxml>=3.8 (from xmlsec==1.3.14)
  Using cached lxml-5.2.1-cp39-cp39-manylinux_2_28_x86_64.whl.metadata (3.4 kB)
Using cached lxml-5.2.1-cp39-cp39-manylinux_2_28_x86_64.whl (5.0 MB)
Building wheels for collected packages: xmlsec
  Building wheel for xmlsec (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for xmlsec (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [39 lines of output]
      WARNING setuptools_scm.pyproject_reading toml section missing 'pyproject.toml does not contain a tool.setuptools_scm section'
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-39
      creating build/lib.linux-x86_64-cpython-39/xmlsec
      copying src/xmlsec/__init__.pyi -> build/lib.linux-x86_64-cpython-39/xmlsec
      copying src/xmlsec/constants.pyi -> build/lib.linux-x86_64-cpython-39/xmlsec
      copying src/xmlsec/template.pyi -> build/lib.linux-x86_64-cpython-39/xmlsec
      copying src/xmlsec/tree.pyi -> build/lib.linux-x86_64-cpython-39/xmlsec
      copying src/xmlsec/py.typed -> build/lib.linux-x86_64-cpython-39/xmlsec
      running build_ext
      building 'xmlsec' extension
      creating build/temp.linux-x86_64-cpython-39
      creating build/temp.linux-x86_64-cpython-39/tmp
      creating build/temp.linux-x86_64-cpython-39/tmp/pip-install-khokr2w2
      creating build/temp.linux-x86_64-cpython-39/tmp/pip-install-khokr2w2/xmlsec_ba08e5441ebd4063854e30606f4e9b9b
      creating build/temp.linux-x86_64-cpython-39/tmp/pip-install-khokr2w2/xmlsec_ba08e5441ebd4063854e30606f4e9b9b/src
      gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DXMLSEC_CRYPTO_DYNAMIC_LOADING=1 -D__XMLSEC_FUNCTION__=__func__ -DXMLSEC_NO_SIZE_T -DXMLSEC_NO_GOST=1 -DXMLSEC_NO_GOST2012=1 -DXMLSEC_DL_LIBLTDL=1 -DMODULE_NAME=xmlsec -DMODULE_VERSION=1.3.14 -I/usr/include/xmlsec1 -I/usr/include/libxml2 -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes/libexslt -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes/libxml -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes/extlibs -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes/libxslt -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes/__pycache__ -I/venv/include -I/opt/python39/include/python3.9 -c /tmp/pip-install-khokr2w2/xmlsec_ba08e5441ebd4063854e30606f4e9b9b/src/constants.c -o build/temp.linux-x86_64-cpython-39/tmp/pip-install-khokr2w2/xmlsec_ba08e5441ebd4063854e30606f4e9b9b/src/constants.o -g -std=c99 -fPIC -fno-strict-aliasing -Wno-error=declaration-after-statement -Werror=implicit-function-declaration -Os
      gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DXMLSEC_CRYPTO_DYNAMIC_LOADING=1 -D__XMLSEC_FUNCTION__=__func__ -DXMLSEC_NO_SIZE_T -DXMLSEC_NO_GOST=1 -DXMLSEC_NO_GOST2012=1 -DXMLSEC_DL_LIBLTDL=1 -DMODULE_NAME=xmlsec -DMODULE_VERSION=1.3.14 -I/usr/include/xmlsec1 -I/usr/include/libxml2 -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes/libexslt -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes/libxml -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes/extlibs -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes/libxslt -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes/__pycache__ -I/venv/include -I/opt/python39/include/python3.9 -c /tmp/pip-install-khokr2w2/xmlsec_ba08e5441ebd4063854e30606f4e9b9b/src/ds.c -o build/temp.linux-x86_64-cpython-39/tmp/pip-install-khokr2w2/xmlsec_ba08e5441ebd4063854e30606f4e9b9b/src/ds.o -g -std=c99 -fPIC -fno-strict-aliasing -Wno-error=declaration-after-statement -Werror=implicit-function-declaration -Os
      gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DXMLSEC_CRYPTO_DYNAMIC_LOADING=1 -D__XMLSEC_FUNCTION__=__func__ -DXMLSEC_NO_SIZE_T -DXMLSEC_NO_GOST=1 -DXMLSEC_NO_GOST2012=1 -DXMLSEC_DL_LIBLTDL=1 -DMODULE_NAME=xmlsec -DMODULE_VERSION=1.3.14 -I/usr/include/xmlsec1 -I/usr/include/libxml2 -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes/libexslt -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes/libxml -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes/extlibs -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes/libxslt -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes/__pycache__ -I/venv/include -I/opt/python39/include/python3.9 -c /tmp/pip-install-khokr2w2/xmlsec_ba08e5441ebd4063854e30606f4e9b9b/src/enc.c -o build/temp.linux-x86_64-cpython-39/tmp/pip-install-khokr2w2/xmlsec_ba08e5441ebd4063854e30606f4e9b9b/src/enc.o -g -std=c99 -fPIC -fno-strict-aliasing -Wno-error=declaration-after-statement -Werror=implicit-function-declaration -Os
      /tmp/pip-install-khokr2w2/xmlsec_ba08e5441ebd4063854e30606f4e9b9b/src/enc.c: In function ‘PyXmlSec_ClearReplacedNodes’:
      /tmp/pip-install-khokr2w2/xmlsec_ba08e5441ebd4063854e30606f4e9b9b/src/enc.c:207:14: warning: assignment to ‘struct LxmlElement **’ from incompatible pointer type ‘PyXmlSec_LxmlElementPtr’ {aka ‘struct LxmlElement *’} [-Wincompatible-pointer-types]
               elem = PyXmlSec_elementFactory(doc, n);
                    ^
      gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DXMLSEC_CRYPTO_DYNAMIC_LOADING=1 -D__XMLSEC_FUNCTION__=__func__ -DXMLSEC_NO_SIZE_T -DXMLSEC_NO_GOST=1 -DXMLSEC_NO_GOST2012=1 -DXMLSEC_DL_LIBLTDL=1 -DMODULE_NAME=xmlsec -DMODULE_VERSION=1.3.14 -I/usr/include/xmlsec1 -I/usr/include/libxml2 -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes/libexslt -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes/libxml -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes/extlibs -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes/libxslt -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes/__pycache__ -I/venv/include -I/opt/python39/include/python3.9 -c /tmp/pip-install-khokr2w2/xmlsec_ba08e5441ebd4063854e30606f4e9b9b/src/exception.c -o build/temp.linux-x86_64-cpython-39/tmp/pip-install-khokr2w2/xmlsec_ba08e5441ebd4063854e30606f4e9b9b/src/exception.o -g -std=c99 -fPIC -fno-strict-aliasing -Wno-error=declaration-after-statement -Werror=implicit-function-declaration -Os
      /tmp/pip-install-khokr2w2/xmlsec_ba08e5441ebd4063854e30606f4e9b9b/src/exception.c: In function ‘PyXmlSec_ExchangeLastError’:
      /tmp/pip-install-khokr2w2/xmlsec_ba08e5441ebd4063854e30606f4e9b9b/src/exception.c:76:9: warning: variable ‘r’ set but not used [-Wunused-but-set-variable]
           int r;
               ^
      gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DXMLSEC_CRYPTO_DYNAMIC_LOADING=1 -D__XMLSEC_FUNCTION__=__func__ -DXMLSEC_NO_SIZE_T -DXMLSEC_NO_GOST=1 -DXMLSEC_NO_GOST2012=1 -DXMLSEC_DL_LIBLTDL=1 -DMODULE_NAME=xmlsec -DMODULE_VERSION=1.3.14 -I/usr/include/xmlsec1 -I/usr/include/libxml2 -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes/libexslt -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes/libxml -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes/extlibs -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes/libxslt -I/tmp/pip-build-env-8vp3i0k9/overlay/lib/python3.9/site-packages/lxml/includes/__pycache__ -I/venv/include -I/opt/python39/include/python3.9 -c /tmp/pip-install-khokr2w2/xmlsec_ba08e5441ebd4063854e30606f4e9b9b/src/keys.c -o build/temp.linux-x86_64-cpython-39/tmp/pip-install-khokr2w2/xmlsec_ba08e5441ebd4063854e30606f4e9b9b/src/keys.o -g -std=c99 -fPIC -fno-strict-aliasing -Wno-error=declaration-after-statement -Werror=implicit-function-declaration -Os
      /tmp/pip-install-khokr2w2/xmlsec_ba08e5441ebd4063854e30606f4e9b9b/src/keys.c: In function ‘PyXmlSec_KeyFromEngine’:
      /tmp/pip-install-khokr2w2/xmlsec_ba08e5441ebd4063854e30606f4e9b9b/src/keys.c:218:65: error: ‘xmlSecKeyDataFormatEngine’ undeclared (first use in this function); did you mean ‘xmlSecKeyDataFormat’?
               key->handle = xmlSecCryptoAppKeyLoad(engine_and_key_id, xmlSecKeyDataFormatEngine, NULL, xmlSecCryptoAppGetDefaultPwdCallback(), (void*)engine_and_key_id);
                                                                       ^~~~~~~~~~~~~~~~~~~~~~~~~
                                                                       xmlSecKeyDataFormat
      /tmp/pip-install-khokr2w2/xmlsec_ba08e5441ebd4063854e30606f4e9b9b/src/keys.c:218:65: note: each undeclared identifier is reported only once for each function it appears in
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for xmlsec
Failed to build xmlsec
ERROR: Could not build wheels for xmlsec, which is required to install pyproject.toml-based projects
  • 👍1
fram-sse commented

I can confirm that this broke my build as well. ubuntu 22.04 on GHA.

shamanthb commented

I can confirm my python project build is broken as well.
Using the docker image: python:3.11-slim-bullseye.

mxamin commented

Thanks for the update, xmlsec1 (C library) introduced breaking changes in v1.3.x which made it hard to support old and new version at the same time. I'll note this issue and hopefully we can fix it in the new release.

@corneauf @jplrank @fram-sse @shamanthb Can you provide the following information so we can debug easier:

  • OS version and distribution
  • xmlsec1 library version
  • libxml2 version
shamanthb commented

@mxamin
Here are the details:
OS: Debian GNU/Linux 11
xmlsec1 version: 1.2.31
libxml2 version: libxml2-dev:arm64 (2.9.10+dfsg-6.7+deb11u4)

  • 👍1
jplrank commented

@mxamin
Here are my details:

  • OS version and distribution: Red Hat Enterprise Linux 8.9 (Ootpa)
  • xmlsec1 library version: 1.2.25-4.el8
  • libxml2 version: 2.9.7-18.el8_9
  • 👍1
fram-sse commented

@mxamin
OS: Ubuntu 20.04.6 LTS
xmlsec1_1.2.28-2_amd64
libxml2_2.9.10+dfsg-5_amd64

  • 👍1
aafaque33 commented

Having same issue in latest installation

Python version: (3.11.6 (main, Oct 27 2023, 00:08:24) [GCC 8.5.0 20210514 (Red Hat 8.5.0-18)])
Operation System: (Rocky Linux 8.8 (Green Obsidian))
xmlsec1 version: 1.2.25
libxml2 version: 2.9.7

jonathangreen commented

For the latest release there are now binary wheels uploaded to pypi for xmlsec. These wheels are compatible with lxml > 5.0.

If you are able to install both xmlsec and lxml from wheels you should be able to install the latest version.

jonathangreen commented

The problem here is that we are not currently compatible with these older libxmlsec1 versions.

If you are able to build against a newer version of libxmlsec1 or use the wheel, which is built against the latest version, then the new release should work.

jimjag commented

Considering that older libxmlsec1 versions do have security issues, and will likely be deprecated, this may resolve itself on its own

bitti commented

Also happens under RHEL9:

OS: Red Hat Enterprise Linux release 9.2 (Plow)
xmlsec1 version: 1.2.29-9.el9
libxml2 version: 2.9.13-6.el9_4

jimjag commented

Can you provide logs of the build error for RHEL9?

jarshwah commented

For the latest release there are now binary wheels uploaded to pypi for xmlsec. These wheels are compatible with lxml > 5.0.

If you are able to install both xmlsec and lxml from wheels you should be able to install the latest version.

I don't think this is right, don't they both compile against different versions of libxml2? Do we know, specifically, which version of libxml2 is used in the xmlsec wheels?

We also use python3-saml which depends on xmlsec, and we were seeing seg faults when using binary wheel versions of xmlsec 1.3.14 and lxml 5.2.2 - see SAML-Toolkits/python3-saml#389 for more details.

I tried installing lxml from source which should use the system libxml2 - and that is compatible with xmlsec 1.3.13 but not with 1.3.14.

So I tried building xmlsec from source @ 1.3.14 and I run into this same compilation failure. So, no matter what I do, I can't seem to have python3-saml + lxml + xmlsec==1.3.14 on the same system.

OS: Linux 6d2916178dc5 6.10.4-linuxkit #1 SMP Wed Oct 2 16:38:00 UTC 2024 aarch64 GNU/Linux
xmlsec1 version: 1.3.14
libxml2 version: 2.9.10+dfsg-6.7+deb11u5
Docker base image: python:3.11.10-slim-bullseye