Pynacl on Apple Silicon/ARM64
johnalanwoods opened this issue · 14 comments
Trying to pip3
install pynacl
on an M1 Mac with pip3
version 20.3.3
.
Get this message on install - any ideas?
Other libs are working ok, such as tinyec
.
Collecting pynacl
Downloading PyNaCl-1.4.0.tar.gz (3.4 MB)
|████████████████████████████████| 3.4 MB 1.8 MB/s
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... error
ERROR: Command errored out with exit status 1:
command: /opt/homebrew/opt/python@3.9/bin/python3.9 /opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/tmpyxshsvrm
cwd: /private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-install-48_63pd2/pynacl_34f24b5a9a4045dfbb49726500e88931
Complete output (39 lines):
Traceback (most recent call last):
File "/opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
main()
File "/opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 133, in prepare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
File "/private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 161, in prepare_metadata_for_build_wheel
self.run_setup()
File "/private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 145, in run_setup
exec(compile(code, __file__, 'exec'), locals())
File "setup.py", line 216, in <module>
setup(
File "/private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
return distutils.core.setup(**attrs)
File "/opt/homebrew/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/core.py", line 108, in setup
_setup_distribution = dist = klass(attrs)
File "/private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 432, in __init__
_Distribution.__init__(self, {
File "/opt/homebrew/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 292, in __init__
self.finalize_options()
File "/private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 708, in finalize_options
ep(self)
File "/private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 715, in _finalize_setup_keywords
ep.load()(self, ep.name, value)
File "/private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/cffi/setuptools_ext.py", line 219, in cffi_modules
add_cffi_module(dist, cffi_module)
File "/private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/cffi/setuptools_ext.py", line 49, in add_cffi_module
execfile(build_file_name, mod_vars)
File "/private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/cffi/setuptools_ext.py", line 25, in execfile
exec(code, glob, glob)
File "src/bindings/build.py", line 36, in <module>
ffi = FFI()
File "/private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/cffi/api.py", line 48, in __init__
import _cffi_backend as backend
ImportError: dlopen(/private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so, 2): Symbol not found: _ffi_prep_closure
Referenced from: /private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so
Expected in: flat namespace
in /private/var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/pip-build-env-btqme7ob/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so
----------------------------------------
ERROR: Command errored out with exit status 1: /opt/homebrew/opt/python@3.9/bin/python3.9 /opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/ws/4b7t11z91pv_wqpqd6dl21x40000gn/T/tmpyxshsvrm Check the logs for full command output.
This is what I have installed:
Package Version
------------ -------
meson 0.56.2
pip 20.3.3
protobuf 3.14.0
pycrypto 2.6.1
pycryptodome 3.9.9
setuptools 51.1.1
six 1.15.0
tinyec 0.3.1
wheel 0.36.2
Building pynacl in a venv using M1 native homebrew python3.9 works on my M1 test machine.
This, unfortunately, makes it highly like that your issue is some type of environment problem specific to your circumstances. The error suggests symbol issues in cffi so you may want to try separately installing cffi first, or reinstalling it using --no-binary cffi to make it recompile in case you have a cached wheel compiled against a non-existent dylib. If you're not installing in a venv try making a new one and installing there as well.
I have the same problem on my M1... bcrypt also fails with exactly the same error.
Tried to install cffi or install it with no-binary, no help...
What happens if you run this command:
python3 -m venv tmpvenv && tmpvenv/bin/pip install -U pip && tmpvenv/bin/pip install pynacl --no-binary :all:
_ffi_prep_closure
is an interesting symbol since it doesn't exist in my symbol table for _cffi_backend.cpython-39-darwin.so
and yet no issues occur on my machine.
The same problem comes up, but it is apparently known issue with CFFI
https://foss.heptapod.net/pypy/cffi/-/issues/464#note_147550
other libs have the same problem, e.g.
gevent/gevent#1721
Maybe it's working for @reaperhulk because he's using x86 arch in terminal/rosetta?
❯ file /usr/local/bin/python3
/usr/local/bin/python3: Mach-O 64-bit executable x86_64
❯ file /opt/homebrew/bin/python3
/opt/homebrew/bin/python3: Mach-O 64-bit executable arm64
yes, I have just setup rosetta2 x86_64 python3 and it works with that one...
@ice29, I guess it will work in time so...
I bought this M1 mac specifically to test against arm64 for PyCA projects so my entire Python toolchain (and everything I use to test pynacl, bcrypt, and cryptography) is M1 native and not run through rosetta 2
❯ file .venv/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so
lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so: Mach-O 64-bit bundle arm64
Given the amount of state inherent in a C build toolchain it's possible I did something (that I've now forgotten) to make this all work I suppose.
@reaperhulk I am unfortunately still getting this error on my m1 mac. I am also in the process of filing a 'downstream' bug report with the Fabric team, as pip install fabric
currently fails on installing pynacl as a dependency due to this bug, fabric
is a developer dependency in my project which means that I'm not presently able to even install my development environment on m1.
Note: I am NOT using rosetta2.
python3 -m venv tmpvenv && tmpvenv/bin/pip install -U pip && tmpvenv/bin/pip install pynacl --no-binary :all:
Collecting pip
Using cached pip-21.1.2-py3-none-any.whl (1.5 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 20.2.3
Uninstalling pip-20.2.3:
Successfully uninstalled pip-20.2.3
Successfully installed pip-21.1.2
Collecting pynacl
Using cached PyNaCl-1.4.0.tar.gz (3.4 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... error
ERROR: Command errored out with exit status 1:
command: /<REDACTED>/bin/python3 /<REDACTED>/tmpvenv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/tmpg27kpjth
cwd: /private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-install-o2vytdsa/pynacl_ded61a19ae59488294e578a090ae4d22
Complete output (39 lines):
Traceback (most recent call last):
File "/<REDACTED>/tmpvenv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 280, in <module>
main()
File "/<REDACTED>/tmpvenv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 263, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/<REDACTED>/tmpvenv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 133, in prepare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
File "/private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 166, in prepare_metadata_for_build_wheel
self.run_setup()
File "/private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 150, in run_setup
exec(compile(code, __file__, 'exec'), locals())
File "setup.py", line 216, in <module>
setup(
File "/private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
return distutils.core.setup(**attrs)
File "/Users/erich/.pyenv/versions/3.9.4/lib/python3.9/distutils/core.py", line 108, in setup
_setup_distribution = dist = klass(attrs)
File "/private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 455, in __init__
_Distribution.__init__(self, {
File "/Users/erich/.pyenv/versions/3.9.4/lib/python3.9/distutils/dist.py", line 292, in __init__
self.finalize_options()
File "/private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 801, in finalize_options
ep(self)
File "/private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 808, in _finalize_setup_keywords
ep.load()(self, ep.name, value)
File "/private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/cffi/setuptools_ext.py", line 219, in cffi_modules
add_cffi_module(dist, cffi_module)
File "/private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/cffi/setuptools_ext.py", line 49, in add_cffi_module
execfile(build_file_name, mod_vars)
File "/private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/cffi/setuptools_ext.py", line 25, in execfile
exec(code, glob, glob)
File "src/bindings/build.py", line 36, in <module>
ffi = FFI()
File "/private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/cffi/api.py", line 48, in __init__
import _cffi_backend as backend
ImportError: dlopen(/private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so, 2): Symbol not found: _ffi_prep_closure
Referenced from: /private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so
Expected in: flat namespace
in /private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-3oog8np4/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so
----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/cf/5a/25aeb636baeceab15c8e57e66b8aa930c011ec1c035f284170cacb05025e/PyNaCl-1.4.0.tar.gz#sha256=54e9a2c849c742006516ad56a88f5c74bf2ce92c9f67435187c3c5953b346505 (from https://pypi.org/simple/pynacl/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*). Command errored out with exit status 1: /<REDACTED>/tmpvenv/bin/python3 /Users/erich/code/work/Project-Vitamin/tmpvenv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/tmpg27kpjth Check the logs for full command output.
@eblume What happens if you uninstall cffi and then attempt to install it with LDFLAGS=-L$(brew --prefix libffi)/lib CFLAGS=-I$(brew --prefix libffi)/include pip install cffi --no-binary :all:
? The problem you're seeing is a mismatch between the libffi cffi compiled against and the one it's finding at runtime. This typically occurs when it compiles against homebrew and then attempts to load the system libffi.
arm64 :: ~
❯ python --version
Python 3.9.4
arm64 :: ~
❯ pip uninstall cffi
Found existing installation: cffi 1.14.5
Uninstalling cffi-1.14.5:
Would remove:
/Users/erich/.pyenv/versions/3.9.4/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so
/Users/erich/.pyenv/versions/3.9.4/lib/python3.9/site-packages/cffi-1.14.5-py3.9.egg-info
/Users/erich/.pyenv/versions/3.9.4/lib/python3.9/site-packages/cffi/*
Proceed (y/n)? y
Successfully uninstalled cffi-1.14.5
arm64 :: ~
❯ LDFLAGS=-L$(brew --prefix libffi)/lib CFLAGS=-I$(brew --prefix libffi)/include pip install cffi --no-binary :all:
Collecting cffi
Using cached cffi-1.14.5.tar.gz (475 kB)
Requirement already satisfied: pycparser in ./.pyenv/versions/3.9.4/lib/python3.9/site-packages (from cffi) (2.20)
Skipping wheel build for cffi, due to binaries being disabled for it.
Installing collected packages: cffi
Running setup.py install for cffi ... done
Successfully installed cffi-1.14.5
arm64 :: ~
❯ python3 -m venv tmpvenv && tmpvenv/bin/pip install -U pip && tmpvenv/bin/pip install pynacl --no-binary :all:
Requirement already satisfied: pip in ./tmpvenv/lib/python3.9/site-packages (21.1.1)
Collecting pip
Using cached pip-21.1.2-py3-none-any.whl (1.5 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 21.1.1
Uninstalling pip-21.1.1:
Successfully uninstalled pip-21.1.1
Successfully installed pip-21.1.2
Collecting pynacl
Using cached PyNaCl-1.4.0.tar.gz (3.4 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... error
ERROR: Command errored out with exit status 1:
command: /Users/erich/tmpvenv/bin/python3 /Users/erich/tmpvenv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/tmp0jl8_sel
cwd: /private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-install-dp3b64jj/pynacl_25a0f9d512d449688f625b9c00c53799
Complete output (39 lines):
Traceback (most recent call last):
<... Redacting Traceback to save some space ...>
ImportError: dlopen(/private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-y_2lilv0/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so, 2): Symbol not found: _ffi_prep_closure
Referenced from: /private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-y_2lilv0/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so
Expected in: flat namespace
in /private/var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/pip-build-env-y_2lilv0/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so
----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/cf/5a/25aeb636baeceab15c8e57e66b8aa930c011ec1c035f284170cacb05025e/PyNaCl-1.4.0.tar.gz#sha256=54e9a2c849c742006516ad56a88f5c74bf2ce92c9f67435187c3c5953b346505 (from https://pypi.org/simple/pynacl/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*). Command errored out with exit status 1: /Users/erich/tmpvenv/bin/python3 /Users/erich/tmpvenv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /var/folders/yv/800l92gj2g99v91mdrzbdjhr0000gn/T/tmp0jl8_sel Check the logs for full command output.
Using cached PyNaCl-1.3.0.tar.gz (3.4 MB)
^C Installing build dependencies ... canceled
ERROR: Operation cancelled by user
I also tried with replacing the last command with simply pip install pynacl
- same effect.
I was able to install using the following setup:
brew install libsodium
ln -s /opt/homebrew/Cellar/libsodium/1.0.18_1/lib/libsodium.a /usr/local/lib/libsodium.a
SODIUM_INSTALL=system LIBSODIUM_MAKE_ARGS=-j8 pip install pynacl
If the previous solution isn't working for you, add CPPFLAGS="-I /opt/homebrew/Cellar/libsodium/1.0.18_1/include"
to the beginning of pip install
.
We ship a universal2 wheel now.