How to run / develop for Feast on M1 Macs
woop opened this issue · 6 comments
Overview
This guide summarizes how both end users and Feast developers can get their M1 Macs set up to use or develop for Feast. pyenv
is used in this guide, but isn't absolutely necessary.
Installation Guide
- Install pyenv
https://github.com/pyenv/pyenv#homebrew-in-macos
- Install dependencies
brew install xz protobuf openssl zlib
- Install Python
(only tested with 3.8.12)
CFLAGS="-I$(brew --prefix xz)/include" LDFLAGS="-L$(brew --prefix xz)/lib" pyenv install 3.8.12
- Upgrade Pip and install dependencies
pip install --upgrade pip
pip install cryptography -U
export CFLAGS="-I$(brew --prefix protobuf)/include"
export LDFLAGS="-L$(brew --prefix protobuf)/lib"
# Please set the protobuf version below (we're hardcoding it to 3.17 here)
pip install protobuf==3.17 --force-reinstall --no-deps --install-option="--cpp_implementation"
You can stop here if you are just a Feast user (not a developer). At this point you should be able to pip install feast
. Steps (5) and (6) are for developers only
- Build and install Feast repository (run from Feast project root)
export LDFLAGS="-L$(brew --prefix freetds)/lib -L$(brew --prefix protobuf)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix openssl)/lib"
export CPPFLAGS="-I$(brew --prefix protobuf)/include -I$(brew --prefix zlib)/include"
export MYSQLCLIENT_LDFLAGS=$(pkg-config --libs mysqlclient)
export MYSQLCLIENT_CFLAGS=$(pkg-config --cflags mysqlclient)
export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1
pip install pip-tools
brew install mysql
make install-python-ci-dependencies PYTHON=3.8
- Run Feast tests (all tests should pass)
FEAST_USAGE=False IS_TEST=True python -m pytest -n 8 sdk/python/tests
Problems & Solutions
Problem
Lack of _ctypes
Solution
Problem
Could not build wheels for cryptography, which is required to install pyproject.toml-based projects
Solution
pip install --upgrade pip
pip install cryptography -U
Problem
distutils.errors.DistutilsExecError: command 'clang' failed with exit status 1
Solution
export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1
Problem
ImportError: cannot import name '_message' from 'google.protobuf.pyext' (/opt/homebrew/Caskroom/miniforge/base/envs/intel_py39/lib/python3.9/site-packages/google/protobuf/pyext/__init__.py)
Solution
brew install protobuf
export CFLAGS="-I$(brew --prefix protobuf)/include"
export LDFLAGS="-L$(brew --prefix protobuf)/lib"
# You need
pip install protobuf==3.17 --force-reinstall --no-deps --install-option="--cpp_implementation"
Problem
ModuleNotFoundError: No module named '_lzma'
Solution
brew install xz
CFLAGS="-I$(brew --prefix xz)/include" LDFLAGS="-L$(brew --prefix xz)/lib" pyenv install 3.9.4
Thanks @woop, it worked like expected.
Just needed to reinstall protobuf as you explained
export CFLAGS="-I$(brew --prefix protobuf)/include"
export LDFLAGS="-L$(brew --prefix protobuf)/lib"
# Please set the protobuf version below (we're hardcoding it to 3.17 here)
pip install protobuf==3.17 --force-reinstall --no-deps --install-option="--cpp_implementation"
Hey @woop using my brand new macbook pro here, M1 chip running Monterey Mac OS. Thank you.
I can confirm this works also with python 3.10.1. Because my pip freeze was empty at the onset of this, I did simplify my protobuf installation to: pip install protobuf --install-option="--cpp_implementation"
.
Hi,
Also, people should use the Rosetta M1 Mac plugin.
I just finished the installation process for the M1 14" Macbook Pro on Mac OSX Monterey 12.1.
Here are the extra steps I had to perform.
Fixing Cryptography Installation
- Add new dependency
brew install rust
- Add these exports before hand:
export CRYPTOGRAPHY_SUPPRESS_LINK_FLAGS="1"
export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include"
- I changed the installation instructions for cryptography to
pip install cryptography==3.4.8 -U
(This is the autogenerated version in the requirements from pip-compile.
Fixing grpc-io installation
- Add these exports before running make.
CFLAGS="-I/opt/homebrew/opt/openssl/include"
LDFLAGS="-L/opt/homebrew/opt/openssl/lib"
Fixing makefile error below
Error: Invalid value for '[SRC_FILES]...': Path 'requirements/py-ci-requirements.txt' does not exist. make: *** [install-python-ci-dependencies] Error 2
- Run
export PYTHON=3.8
Another issue that I found with setup with java. The protobuf and protc-gen-grpc-java artifacts are not yet available for m1 arm mac osx. In order to fix the artifact not found error when you build make build-java
, make test-java
and make test-java-integration
, change the two artifacts in java/datatypes/pom.xml
to
<protocArtifact>
com.google.protobuf:protoc:${protoc.version}:exe:osx-x86_64
</protocArtifact>
<pluginId>grpc-java</pluginId>
<pluginArtifact>
io.grpc:protoc-gen-grpc-java:${grpc.version}:exe:osx-x86_64
</pluginArtifact>
You might have to change backports-zoneinfo==0.2.1 to backports-zoneinfo==0.2.1;python_version<"3.9" in sdk/python/requirements/py3.8-ci-requirements.txt to make sure it doesn't grab an unworkable copy of backports-zoneinfo.
In the most recent pip 24.0
release, the --install-option
has been deprecated. As an alternative, you can use
--config-settings
in place of --install-option
.
Here's the modified the command accordingly:
pip install protobuf --force-reinstall --no-deps --config-settings="--build-option=--cpp_implementation"