Unable to install under M1 Macintosh mode (works in x86/rosetta 2 mode)
bschollnick opened this issue · 11 comments
I am unable to install pymupdf under Mac OS X Big Sur Native M1 (eg pip install PyMuPDF), whereas in Intel mode (Rosetta 2/aka Intel translation mode) it works without any issue.
(quickbbs) benjamin@Nerv quickbbs % pip install pymupdf
Collecting pymupdf
Using cached PyMuPDF-1.18.6.tar.gz (258 kB)
Using legacy 'setup.py install' for pymupdf, since package 'wheel' is not installed.
Installing collected packages: pymupdf
Running setup.py install for pymupdf ... error
ERROR: Command errored out with exit status 1:
command: /Users/benjamin/venvs/quickbbs/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/6g/0vldjqgd5d55rdv163phqh6r0000gn/T/pip-install-dqerqvac/pymupdf_74020f613ca84fc09a79e826a22842db/setup.py'"'"'; __file__='"'"'/private/var/folders/6g/0vldjqgd5d55rdv163phqh6r0000gn/T/pip-install-dqerqvac/pymupdf_74020f613ca84fc09a79e826a22842db/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/6g/0vldjqgd5d55rdv163phqh6r0000gn/T/pip-record-lq0xtd43/install-record.txt --single-version-externally-managed --compile --install-headers /Users/benjamin/venvs/quickbbs/include/site/python3.9/pymupdf
cwd: /private/var/folders/6g/0vldjqgd5d55rdv163phqh6r0000gn/T/pip-install-dqerqvac/pymupdf_74020f613ca84fc09a79e826a22842db/
Complete output (20 lines):
running install
running build
running build_py
creating build
creating build/lib.macosx-11-arm64-3.9
creating build/lib.macosx-11-arm64-3.9/fitz
copying fitz/__init__.py -> build/lib.macosx-11-arm64-3.9/fitz
copying fitz/fitz.py -> build/lib.macosx-11-arm64-3.9/fitz
copying fitz/utils.py -> build/lib.macosx-11-arm64-3.9/fitz
copying fitz/__main__.py -> build/lib.macosx-11-arm64-3.9/fitz
running build_ext
building 'fitz._fitz' extension
creating build/temp.macosx-11-arm64-3.9
creating build/temp.macosx-11-arm64-3.9/fitz
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/opt/homebrew/include -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/opt/homebrew/opt/openssl@1.1/include -I/usr/local/include/mupdf -I/usr/local/include -Imupdf/thirdparty/freetype/include -I/opt/homebrew/include -I/opt/homebrew/opt/openssl@1.1/include -I/opt/homebrew/opt/sqlite/include -I/opt/homebrew/opt/tcl-tk/include -I/Users/benjamin/venvs/quickbbs/include -I/opt/homebrew/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c fitz/fitz_wrap.c -o build/temp.macosx-11-arm64-3.9/fitz/fitz_wrap.o
fitz/fitz_wrap.c:2767:10: fatal error: 'fitz.h' file not found
#include <fitz.h>
^~~~~~~~
1 error generated.
error: command '/usr/bin/clang' failed with exit code 1
----------------------------------------
ERROR: Command errored out with exit status 1: /Users/benjamin/venvs/quickbbs/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/6g/0vldjqgd5d55rdv163phqh6r0000gn/T/pip-install-dqerqvac/pymupdf_74020f613ca84fc09a79e826a22842db/setup.py'"'"'; __file__='"'"'/private/var/folders/6g/0vldjqgd5d55rdv163phqh6r0000gn/T/pip-install-dqerqvac/pymupdf_74020f613ca84fc09a79e826a22842db/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/6g/0vldjqgd5d55rdv163phqh6r0000gn/T/pip-record-lq0xtd43/install-record.txt --single-version-externally-managed --compile --install-headers /Users/benjamin/venvs/quickbbs/include/site/python3.9/pymupdf Check the logs for full command output.
It appears that it is being built and the wheel is being ignored.
Your configuration (mandatory)
-
MAc OS X Big Sur, M1 Mac Mini w/16 Gb Ram
-
Python 3.9.1 (default, Jan 8 2021, 12:11:08)
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin -
PyMuPDF Using cached PyMuPDF-1.18.6.tar.gz (258 kB)
Additional context (optional)
I suspect that the Universal 2 wheel just needs to be created. Cryptography, and Pillow within the last day, just started to work in M1 Native mode, due to Wheel support being added to the build systems...
If there is anything that I can assist with or help, please feel free to let me know. I'd be happy to assist.
It pip tries to install from sources like it does here, then the platform is not supported by any of the existing wheels.
Is that a 32bit Python?
Ah no, it seems an ARM64 architecture on Mac.
Well, this is not supportable by the wheel generation serivice I am using (Travis CI). It also isn't by the other services.
So it seems you have to generate from sources.
And that means install MuPDF first (missing this leads to fitz.h
not found), then PyMuPDF as described on the repo home page ...
Hope you can make your way - I won't be able to help here.
If I am understanding what little documentation I have looked at regarding Travis CI.
Xcode 12.0 is available by adding osx_image: xcode12u to your .travis.yml.
Xcode 12.2 is available by adding osx_image: xcode12.2 to your .travis.yml.
And Xcode 12/12.2 support is what is needed for Universal support for M1 ARM... I believe. But I don't know if the travis.yml file is at Github's side, or a feature you directly control?
Could you check and see if that's available, I would greatly appreciate it.
I can check what happens when I add that osx_image
statement to myyml file.
There currently is none, so xcode 9.4.1 is taken. Maybe by cibuildwheel
... let me check.
Ok, made these changes to my ravis yml:
It worked - here is the the run log from Travis CI, which shows xcode 12.2 is indeed used.
OSX-runlog.zip
The generated wheels are here: https://github.com/pymupdf/PyMuPDF-wheels/tree/osx
They have the platform tag ...-macosx_10_9_x86_64.whl
- which is not any different from before. I know that pip determines wheel compatibility by simply looking at the wheel filename / platform tag, so I am afraid it will not react any differently 😟.
If that concern confirms ... I don't know.
- rename the wheel file?
- manually unzip and copy into Python folder?
- convert wheel back to egg format?
- I tested in Rosetta2 (aka Universal 2), and it still loads fine, and works.
- It indeed does fail under M1 Native. Renaming the Wheel was necessary, but if imported, complains about the wrong architecture (eg. X86 vs Arm64).
So just adding the compile tag for Travis CI obviously isn't the whole story...
I did some more digging, and I wish to stress, this is the first time I've even looked at Travis, beyond seeing travis logs from builds in a few repositories...
I know that it does work after being updated, because I was testing this for Pillow, and psycopg2. My understanding is that they updated their Travis CI builds to add ARM64 support.
This is a travis ci build YAML from Microsoft (LWJGL/lwjgl3#601 (comment)),
https://github.com/LWJGL-CI/glfw/blob/c5ee7da34cad750429b1f2bea4c6ea7ed4e7de6f/.travis.yml#L305
The major differences that I see are the cmake parameters (eg -DCMAKE_OSX_DEPLOYMENT_TARGET=10.9 vs -DCMAKE_OSX_DEPLOYMENT_TARGET=11.0) and the addition of the cmake architectures.
I'm just surprised that I can't find any clear cut documentation on how to add this to Travis CI....
The appveyor.yml also was updated in that same commit to add "PLATFORM: ARM64", so maybe that also needs to be updated? https://github.com/LWJGL-CI/glfw/blob/c5ee7da34cad750429b1f2bea4c6ea7ed4e7de6f/appveyor.yml#L19
I just want to stress, I'll be happy to continue testing as needed.
# -----------------
- name: "macOS x64"
# -----------------
os: osx
osx_image: xcode12.2
compiler: clang
install:
- brew install awscli
script:
- mkdir build
- cd build
- "cmake $GLFW_PARAMS -DCMAKE_OSX_DEPLOYMENT_TARGET=10.9 .."
- cmake --build .
- strip -u -r src/libglfw.dylib
- "git log --first-parent --pretty=format:%H HEAD~2..HEAD~1 > libglfw.dylib.git"
- "aws s3 cp src/libglfw.dylib s3://build.lwjgl.org/nightly/macosx/x64/ $S3_PARAMS"
- "aws s3 cp libglfw.dylib.git s3://build.lwjgl.org/nightly/macosx/x64/ $S3_PARAMS"
# -----------------
- name: "macOS arm64"
# -----------------
os: osx
osx_image: xcode12.2
compiler: clang
install:
- brew install awscli
script:
- mkdir build
- cd build
- "cmake $GLFW_PARAMS -DCMAKE_OSX_DEPLOYMENT_TARGET=11.0 -DCMAKE_OSX_ARCHITECTURES=arm64 .."
- cmake --build .
- strip -u -r src/libglfw.dylib
- "git log --first-parent --pretty=format:%H HEAD~2..HEAD~1 > libglfw.dylib.git"
- "aws s3 cp src/libglfw.dylib s3://build.lwjgl.org/nightly/macosx/arm64/ $S3_PARAMS"
- "aws s3 cp libglfw.dylib.git s3://build.lwjgl.org/nightly/macosx/arm64/ $S3_PARAMS"
I am a little lost at the moment.
The examples you mentioned do not use cibuildwheel like PyMuPDF. cibuildwheel has a set of supported target architectures platform tags - see here.
Reading some info about Rosetta2 I learned that this is some feature that supports x86 apps made for Intel processors to run on M1. It is an emulation.
I do not even know whether MuPDF can run / be generated on / for x64 M1 ...
Yeah, I have read this: register QEMU via binfmt_misc
but couldn't figure out what I have to do. Just setting the option CIBW_ARCHS_LINUX
to some value is not enough definitely.
If the new Python is downloaded, then the trick linked previously is used to force x86_64 or arm64 as specified.
The main issue I see, which echo’s my issues earlier, is to insure you have the latest PIP, packaging, wheel modules. They all have upgrades need to support the M1.
All this is covered in one go, when the wheel generation requets is sent to Travic CI. The numerous options and "export" variables, like the one mentioned above, just control the amount of different wheels generated.
This includes taking care of the different Pythons with their latest PIPs and other stuff being installed when the Travis virtual machines are being activated.
Yeah, I have read this:
register QEMU via binfmt_misc
but couldn't figure out what I have to do. Just setting the optionCIBW_ARCHS_LINUX
to some value is not enough definitely.
On Linux with docker installed, registering qemu via binfmt_misc can be as easy as:
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
no idea if that works on OS X though. Might still be helpful to have arm wheels for linux though.