apache/datasketches-cpp

The Python package for ARM MacOS has an x86_64 datasketches.so in it

Closed this issue · 4 comments

andrey-klochkov-liftoff commented

When I install datasketches 3.5.0 with pip and the try to import it, I get the following on an ARM based MacBook:

>>> import datasketches
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(/Users/username/venv/lib/python3.9/site-packages/datasketches.so, 0x0002): tried: '/Users/username/venv/lib/python3.9/site-packages/datasketches.so' (mach-o file, but is an incompatible architecture (have (x86_64), need (arm64e)))

I downloaded the file datasketches-3.5.0-cp39-cp39-macosx_11_0_arm64.whl from pypi.org, extracted datasketches.so from it and then checked the platform, and it's a wrong platform:

❯ file datasketches.so
datasketches.so: Mach-O 64-bit bundle x86_64
jmalkin commented

Well, that certainly seems broken. Ugh. Thanks for flagging it.

I have an M1 Mac as of this week so I'll see if I can figure out what's happening -- and even be able to test it directly, finally!

andrey-klochkov-liftoff commented

We just built the wheel from sources on an arm macbook, installed it and it worked fine. Seems something's off with the way those pypi packages were built, not the library code.

jmalkin commented

Finally had a chance to look into this. Found an explanation from the same issue on other projects. The wheels are built via GitHub Actions, which apparently uses only Intel Macs as the actual task runners. The environment is mostly set to cross-compile -- but cmake is unaware of that when it configures the underlying C++ library.

That also explains why building a wheel locally works fine. Having a complete theory of the problem is good.

I'll try to test a patch later tonight.

jmalkin commented

At long last, this has been resolved. Pypi now has v3.5.1, which fixes this. At least it worked when I tested it just now.

Apologies for the delay. The issue was very much not forgotten.

  • 👍1