msgpack/msgpack-python

msgpack's C extension is not built.

ThomasWaldmann opened this issue · 8 comments

ThomasWaldmann commented

I am seeing this failure on netbsd since 1.0.8 (test works with 1.0.7) - maybe due to the recent changes in msgpack packaging?:

    def is_slow_msgpack():
        import msgpack
        import msgpack.fallback
        return msgpack.Packer is msgpack.fallback.Packer


    def test_is_slow_msgpack():
        # we need to import upstream msgpack package here, not helpers.msgpack:
        import msgpack
        import msgpack.fallback
        saved_packer = msgpack.Packer
        try:
            msgpack.Packer = msgpack.fallback.Packer
            assert is_slow_msgpack()
        finally:
            msgpack.Packer = saved_packer
        # this tests that we have fast msgpack on test platform:
>       assert not is_slow_msgpack()
E       assert not True
E        +  where True = is_slow_msgpack()

.tox/py311/lib/python3.11/site-packages/borg/testsuite/helpers.py:726: AssertionError

The msgpack packages get installed via pip.

ThomasWaldmann commented

1.0.8:

(borg-env) [vagrant@netbsd9 borg]$ pip install -v 'msgpack==1.0.8'
Using pip 23.3.2 from /vagrant/borg/borg-env/lib/python3.11/site-packages/pip (python 3.11)
Collecting msgpack==1.0.8
  Obtaining dependency information for msgpack==1.0.8 from https://files.pythonhosted.org/packages/ce/39/3a1f468109c02d8c3780d0731555f05e0b332152e1ce2582c2f97ab9ff25/msgpack-1.0.8-py3-none-any.whl.metadata
  Downloading msgpack-1.0.8-py3-none-any.whl.metadata (9.1 kB)
Downloading msgpack-1.0.8-py3-none-any.whl (15 kB)
Installing collected packages: msgpack
Successfully installed msgpack-1.0.8
methane commented

Yes. See #576. I shipped pure Python wheel.

ThomasWaldmann commented

1.0.7:

(borg-env) [vagrant@netbsd9 borg]$ pip install -v 'msgpack==1.0.7'
Using pip 23.3.2 from /vagrant/borg/borg-env/lib/python3.11/site-packages/pip (python 3.11)
Collecting msgpack==1.0.7
  Downloading msgpack-1.0.7.tar.gz (166 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 166.3/166.3 kB 1.3 MB/s eta 0:00:00
  Running command pip subprocess to install build dependencies
  Collecting Cython~=3.0.0
    Downloading Cython-3.0.9-py2.py3-none-any.whl.metadata (3.2 kB)
  Collecting setuptools>=35.0.2
    Downloading setuptools-69.1.1-py3-none-any.whl.metadata (6.2 kB)
  Downloading Cython-3.0.9-py2.py3-none-any.whl (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 1.4 MB/s eta 0:00:00
  Downloading setuptools-69.1.1-py3-none-any.whl (819 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 819.3/819.3 kB 1.5 MB/s eta 0:00:00
  Installing collected packages: setuptools, Cython
  Successfully installed Cython-3.0.9 setuptools-69.1.1

  [notice] A new release of pip is available: 23.3.2 -> 24.0
  [notice] To update, run: pip install --upgrade pip
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  running egg_info
  writing msgpack.egg-info/PKG-INFO
  writing dependency_links to msgpack.egg-info/dependency_links.txt
  writing top-level names to msgpack.egg-info/top_level.txt
  reading manifest file 'msgpack.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching '*.c' under directory 'msgpack'
  adding license file 'COPYING'
  writing manifest file 'msgpack.egg-info/SOURCES.txt'
  Getting requirements to build wheel ... done
  Running command pip subprocess to install backend dependencies
  Collecting wheel
    Downloading wheel-0.42.0-py3-none-any.whl.metadata (2.2 kB)
  Downloading wheel-0.42.0-py3-none-any.whl (65 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.4/65.4 kB 876.3 kB/s eta 0:00:00
  Installing collected packages: wheel
  Successfully installed wheel-0.42.0

  [notice] A new release of pip is available: 23.3.2 -> 24.0
  [notice] To update, run: pip install --upgrade pip
  Installing backend dependencies ... done
  Running command Preparing metadata (pyproject.toml)
  running dist_info
  creating /tmp/pip-modern-metadata-ep9xgs27/msgpack.egg-info
  writing /tmp/pip-modern-metadata-ep9xgs27/msgpack.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-ep9xgs27/msgpack.egg-info/dependency_links.txt
  writing top-level names to /tmp/pip-modern-metadata-ep9xgs27/msgpack.egg-info/top_level.txt
  writing manifest file '/tmp/pip-modern-metadata-ep9xgs27/msgpack.egg-info/SOURCES.txt'
  reading manifest file '/tmp/pip-modern-metadata-ep9xgs27/msgpack.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching '*.c' under directory 'msgpack'
  adding license file 'COPYING'
  writing manifest file '/tmp/pip-modern-metadata-ep9xgs27/msgpack.egg-info/SOURCES.txt'
  creating '/tmp/pip-modern-metadata-ep9xgs27/msgpack-1.0.7.dist-info'
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: msgpack
  Running command Building wheel for msgpack (pyproject.toml)
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.netbsd-9.3-amd64-cpython-311
  creating build/lib.netbsd-9.3-amd64-cpython-311/msgpack
  copying msgpack/__init__.py -> build/lib.netbsd-9.3-amd64-cpython-311/msgpack
  copying msgpack/exceptions.py -> build/lib.netbsd-9.3-amd64-cpython-311/msgpack
  copying msgpack/ext.py -> build/lib.netbsd-9.3-amd64-cpython-311/msgpack
  copying msgpack/fallback.py -> build/lib.netbsd-9.3-amd64-cpython-311/msgpack
  running egg_info
  writing msgpack.egg-info/PKG-INFO
  writing dependency_links to msgpack.egg-info/dependency_links.txt
  writing top-level names to msgpack.egg-info/top_level.txt
  reading manifest file 'msgpack.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching '*.c' under directory 'msgpack'
  adding license file 'COPYING'
  writing manifest file 'msgpack.egg-info/SOURCES.txt'
  copying msgpack/_cmsgpack.cpp -> build/lib.netbsd-9.3-amd64-cpython-311/msgpack
  copying msgpack/_cmsgpack.pyx -> build/lib.netbsd-9.3-amd64-cpython-311/msgpack
  copying msgpack/_packer.pyx -> build/lib.netbsd-9.3-amd64-cpython-311/msgpack
  copying msgpack/_unpacker.pyx -> build/lib.netbsd-9.3-amd64-cpython-311/msgpack
  copying msgpack/buff_converter.h -> build/lib.netbsd-9.3-amd64-cpython-311/msgpack
  copying msgpack/pack.h -> build/lib.netbsd-9.3-amd64-cpython-311/msgpack
  copying msgpack/pack_template.h -> build/lib.netbsd-9.3-amd64-cpython-311/msgpack
  copying msgpack/sysdep.h -> build/lib.netbsd-9.3-amd64-cpython-311/msgpack
  copying msgpack/unpack.h -> build/lib.netbsd-9.3-amd64-cpython-311/msgpack
  copying msgpack/unpack_define.h -> build/lib.netbsd-9.3-amd64-cpython-311/msgpack
  copying msgpack/unpack_template.h -> build/lib.netbsd-9.3-amd64-cpython-311/msgpack
  running build_ext
  building 'msgpack._cmsgpack' extension
  creating build/temp.netbsd-9.3-amd64-cpython-311
  creating build/temp.netbsd-9.3-amd64-cpython-311/msgpack
  gcc -pthread -Wsign-compare -DNDEBUG -O2 -I/usr/include -I/usr/pkg/include -O2 -I/usr/include -I/usr/pkg/include -O2 -I/usr/include -I/usr/pkg/include -fPIC -I. -I/vagrant/borg/borg-env/include -I/usr/pkg/include/python3.11 -c msgpack/_cmsgpack.cpp -o build/temp.netbsd-9.3-amd64-cpython-311/msgpack/_cmsgpack.o
  c++ -pthread -shared -Wl,-zrelro -L/usr/lib -Wl,-R/usr/lib -L/usr/pkg/lib -Wl,-R/usr/pkg/lib -L/usr/X11R7/lib -Wl,-R/usr/X11R7/lib -Wl,-zrelro -L/usr/lib -Wl,-R/usr/lib -L/usr/pkg/lib -Wl,-R/usr/pkg/lib -L/usr/X11R7/lib -Wl,-R/usr/X11R7/lib build/temp.netbsd-9.3-amd64-cpython-311/msgpack/_cmsgpack.o -L/usr/pkg/lib -o build/lib.netbsd-9.3-amd64-cpython-311/msgpack/_cmsgpack.so
  installing to build/bdist.netbsd-9.3-amd64/wheel
  running install
  running install_lib
  creating build/bdist.netbsd-9.3-amd64
  creating build/bdist.netbsd-9.3-amd64/wheel
  creating build/bdist.netbsd-9.3-amd64/wheel/msgpack
  copying build/lib.netbsd-9.3-amd64-cpython-311/msgpack/__init__.py -> build/bdist.netbsd-9.3-amd64/wheel/msgpack
  copying build/lib.netbsd-9.3-amd64-cpython-311/msgpack/exceptions.py -> build/bdist.netbsd-9.3-amd64/wheel/msgpack
  copying build/lib.netbsd-9.3-amd64-cpython-311/msgpack/ext.py -> build/bdist.netbsd-9.3-amd64/wheel/msgpack
  copying build/lib.netbsd-9.3-amd64-cpython-311/msgpack/fallback.py -> build/bdist.netbsd-9.3-amd64/wheel/msgpack
  copying build/lib.netbsd-9.3-amd64-cpython-311/msgpack/_cmsgpack.cpp -> build/bdist.netbsd-9.3-amd64/wheel/msgpack
  copying build/lib.netbsd-9.3-amd64-cpython-311/msgpack/_cmsgpack.pyx -> build/bdist.netbsd-9.3-amd64/wheel/msgpack
  copying build/lib.netbsd-9.3-amd64-cpython-311/msgpack/_packer.pyx -> build/bdist.netbsd-9.3-amd64/wheel/msgpack
  copying build/lib.netbsd-9.3-amd64-cpython-311/msgpack/_unpacker.pyx -> build/bdist.netbsd-9.3-amd64/wheel/msgpack
  copying build/lib.netbsd-9.3-amd64-cpython-311/msgpack/buff_converter.h -> build/bdist.netbsd-9.3-amd64/wheel/msgpack
  copying build/lib.netbsd-9.3-amd64-cpython-311/msgpack/pack.h -> build/bdist.netbsd-9.3-amd64/wheel/msgpack
  copying build/lib.netbsd-9.3-amd64-cpython-311/msgpack/pack_template.h -> build/bdist.netbsd-9.3-amd64/wheel/msgpack
  copying build/lib.netbsd-9.3-amd64-cpython-311/msgpack/sysdep.h -> build/bdist.netbsd-9.3-amd64/wheel/msgpack
  copying build/lib.netbsd-9.3-amd64-cpython-311/msgpack/unpack.h -> build/bdist.netbsd-9.3-amd64/wheel/msgpack
  copying build/lib.netbsd-9.3-amd64-cpython-311/msgpack/unpack_define.h -> build/bdist.netbsd-9.3-amd64/wheel/msgpack
  copying build/lib.netbsd-9.3-amd64-cpython-311/msgpack/unpack_template.h -> build/bdist.netbsd-9.3-amd64/wheel/msgpack
  copying build/lib.netbsd-9.3-amd64-cpython-311/msgpack/_cmsgpack.so -> build/bdist.netbsd-9.3-amd64/wheel/msgpack
  running install_egg_info
  Copying msgpack.egg-info to build/bdist.netbsd-9.3-amd64/wheel/msgpack-1.0.7-py3.11.egg-info
  running install_scripts
  creating build/bdist.netbsd-9.3-amd64/wheel/msgpack-1.0.7.dist-info/WHEEL
  creating '/tmp/pip-wheel-vvcbp817/.tmp-cdhlzada/msgpack-1.0.7-cp311-cp311-netbsd_9_3_amd64.whl' and adding 'build/bdist.netbsd-9.3-amd64/wheel' to it
  adding 'msgpack/__init__.py'
  adding 'msgpack/_cmsgpack.cpp'
  adding 'msgpack/_cmsgpack.pyx'
  adding 'msgpack/_cmsgpack.so'
  adding 'msgpack/_packer.pyx'
  adding 'msgpack/_unpacker.pyx'
  adding 'msgpack/buff_converter.h'
  adding 'msgpack/exceptions.py'
  adding 'msgpack/ext.py'
  adding 'msgpack/fallback.py'
  adding 'msgpack/pack.h'
  adding 'msgpack/pack_template.h'
  adding 'msgpack/sysdep.h'
  adding 'msgpack/unpack.h'
  adding 'msgpack/unpack_define.h'
  adding 'msgpack/unpack_template.h'
  adding 'msgpack-1.0.7.dist-info/COPYING'
  adding 'msgpack-1.0.7.dist-info/METADATA'
  adding 'msgpack-1.0.7.dist-info/WHEEL'
  adding 'msgpack-1.0.7.dist-info/top_level.txt'
  adding 'msgpack-1.0.7.dist-info/RECORD'
  removing build/bdist.netbsd-9.3-amd64/wheel
  Building wheel for msgpack (pyproject.toml) ... done
  Created wheel for msgpack: filename=msgpack-1.0.7-cp311-cp311-netbsd_9_3_amd64.whl size=236700 sha256=b3f23a386970c87442f34c68fd7c9a0210753a64fb5316cff4548464c7b06e6c
  Stored in directory: /home/vagrant/.cache/pip/wheels/50/3e/01/1b909b1529ece55ec94c07614a9d95421f2d62d7932a429e47
Successfully built msgpack
Installing collected packages: msgpack
Successfully installed msgpack-1.0.7
methane commented

You can avoid pure Python wheel by ---no-binary :all:.

  • 👍1
ThomasWaldmann commented

The pure python wheel would slow down borgbackup significantly, I guess.

So, guess the "solution" is to manually run this before borg install:

$ pip install -v --no-binary msgpack 'msgpack==1.0.8'

Quite sure many people won't do that.

ThomasWaldmann commented

Same issue on freebsd, openbsd, ...

jfolz commented

@ThomasWaldmann what if you add --no-binary=msgpack to the requirements?

methane commented

I removed "none-any" wheel from PyPI.

  • 👍1