Error importing gdstk >= 0.9.38 on Mac M2
Closed this issue · 21 comments
between 0.9.37 and 0.9.38, I am not able to import gdstk properly on my Mac M2.
Must be an issue with something in here
I tried re-installing cmake and zlib with brew, no luck.
>>> import gdstk
ImportError: dlopen(/Users/twhughes/.pyenv/versions/3.10.9/lib/python3.10/site-packages/gdstk.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace (__ZN5gdstk12error_loggerE)
Note: running MacOS 12.5.1 Monterey
one workwaround is to conda install gdstk
with pip on m1/m2 there are issues, it would be great to fix this
I faced the same issue...
I am new to python so not aware how the packaging works for native builds.
if some1 can guide me with the process i can create a PR which works for m1/m2 chips
@tylerflex were you able to find a solution to this issue? I am facing the same problem on my Mac M1.
How about using?
conda install gdstk
that will work except for the latest gdstk version, which we haven't been able to release in conda
The 'conda install gdstk' method worked for me!
Would be great to fix this issue, I use homebrew and not conda to manage all my Python development environments and don't want to switch just for this. I'm having the same issue on my M2 MacBook Pro running Ventura 13.6.
This is an ARM-architecture specific problem. I have both the ARM and Intel versions of homebrew (and Python) installed, and the wheel for the Intel versions, which run through Rosetta, work fine. But the ARM version produces the following error:
>>> import gdstk
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/sploeg/envs/temp/lib/python3.11/site-packages/gdstk/__init__.py", line 1, in <module>
from .gdstk import *
ImportError: dlopen(/Users/sploeg/envs/temp/lib/python3.11/site-packages/gdstk/gdstk.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '__ZN5gdstk12error_loggerE'
After running 'conda install gdstk' I do not get the dlopen ImportError when importing gdstk in my systems current version of python (3.11.6), however I am still getting the error in python 3.9. I am trying to resolve this error specifically for python3.9 because I want to import gdsfactory into klayout (on M1 Mac), which uses python 3.9, and I am getting this same ImportError when trying to import gdsfactory in klayout or python3.9 through the terminal. I have created a new conda environment that uses python 3.9 and conda installed gdstk + pip installed gdsfactory, and am able to import gdsfactory successfully here and write gdsfactory code. However, once I exit out of this virtual env, I am still unable to import gdsfactory in python3.9 through the terminal or klayout. Does anyone have suggestions on how I could resolve this? Thanks!
Update: klayout's python3.9 (and my system python 3.9) are managed by Homebrew, whereas my systems python3.11 is managed by conda, so that's why these conda updates aren't applying to them.
any updates on this?
I found this the other day: https://blog.tim-smith.us/2015/09/python-extension-modules-os-x/ but I'm not sure if it would help or if it's something that's already taken care of (it's from 2015). In any case, without an m2, I can't do much debugging. If anyone with a m2 is willing to spend some hours figuring this out, please do.
Can someone with a MAC test this? :)
Maybe this helps in some way, so apparently Github Actions now have beta M1 public runners. Potentially we can set up a test to reproduce this issue that mirrors the conda recipe one but changing the build configuration with Lucas's suggestion potentially modifying the testing Github action should trigger this to verify if any patch works. We wouldn't have to pin gdstk for tidy3d in that case?
I might be able to give this a try tomorrow using the github M1 runners. I'll post any updates here!
I am not sure what to test here. I currently have a working (at least I think so) gdstk installed via anaconda from conda-forge's channel. I'm on MacOS 14.1 with M2 chip. (I don't have Rosetta yet. Powering through.)
> conda list
gdstk 0.9.46 py311h52bd988_0 conda-forge
...
conda search -c conda-forge gdstk==0.9.46 --info
...
gdstk 0.9.46 py311h52bd988_0
----------------------------
file name : gdstk-0.9.46-py311h52bd988_0.conda
name : gdstk
version : 0.9.46
build : py311h52bd988_0
build number: 0
size : 306 KB
license : BSL-1.0
subdir : osx-arm64
url : https://conda.anaconda.org/conda-forge/osx-arm64/gdstk-0.9.46-py311h52bd988_0.conda
md5 : c296a8273195c56fe6fb387d1d67ed6b
timestamp : 2023-10-13 20:20:58 UTC
dependencies:
- __osx >=10.9
- libcxx >=16.0.6
- libzlib >=1.2.13,<1.3.0a0
- numpy >=1.23.5,<2.0a0
- python >=3.11,<3.12.0a0
- python >=3.11,<3.12.0a0 *_cpython
- python_abi 3.11.* *_cp311
- qhull >=2020.2,<2020.3.0a0
@tylerflex I think your error means that the gdstk .so file is not finding a dynamic library. To help with debug, here's the dynamic libraries loaded by gdstk on my system. Would be nice to check if you have the same versions (and if they are installed). Mine were all managed by conda That's where I would start looking:
❯ python -c "import gdstk; print(gdstk.__file__)"
/Users/thomas/miniconda3/envs/mp/lib/python3.11/site-packages/gdstk/__init__.py
❯ otool -L /Users/thomas/miniconda3/envs/mp/lib/python3.11/site-packages/gdstk/gdstk.cpython-311-darwin.so
/Users/thomas/miniconda3/envs/mp/lib/python3.11/site-packages/gdstk/gdstk.cpython-311-darwin.so:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.0.0)
@rpath/libz.1.dylib (compatibility version 1.0.0, current version 1.2.13)
@rpath/libqhull_r.8.0.dylib (compatibility version 8.0.0, current version 8.0.2)
@rpath/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
❯ conda list "gdstk|libcxx|libzlib|qhull"
# packages in environment at /Users/thomas/miniconda3/envs/mp:
#
# Name Version Build Channel
gdstk 0.9.46 py311h52bd988_0 conda-forge
libcxx 16.0.6 h4653b0c_0 conda-forge
libzlib 1.2.13 h53f4e23_5 conda-forge
qhull 2020.2 hc021e02_2 conda-forge
I'm also hijacking this comment to see if anyone is interested in developing a klayout.app homebrew formula with me (essentially the LW version from Kazzz). I'm running a setup like this now and it works fine without rosetta.
Hi Thomas,
yes, using conda it works,
some companies are forbidden to use conda and they would need to pip install gdstk with the brew python from Mac
I've added a fix and enabled arm64 wheel builds in release v0.9.48.
When installing, you might want to use the pip flag --only-binary if it tries to compile from source. Compiling from source should also work, but you need to have all C++ dependencies in place.
If anyone with Apple Silicon give this a try, let me know if it works! @tylerflex @sequoiap @jasminabrar
Thanks @heitzmann !
I was able to successfully pip install it, but then when importing the package I get this.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/twhughes/.pyenv/versions/3.10.9/lib/python3.10/site-packages/gdstk/__init__.py", line 1, in <module>
from .gdstk import *
ImportError: dlopen(/Users/twhughes/.pyenv/versions/3.10.9/lib/python3.10/site-packages/gdstk/gdstk.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '__ZN5gdstk12error_loggerE'
>>>
which I think was the original error I was getting
Thanks @tylerflex! I'm thinking that the issue is still the cross-compilation (we're using the intel MacOS runners to compile the arm64 architecture). The python interface might be working correctly now, but the internal library might still have issues. I have to figure out how to instruct the build system to also cross-compile libgdstk.a.
Another release (0.9.49) is live on pypi with a possible fix for Apple Silicon. Now it uses a universal wheel that is supposed to include both architectures. Please make sure you're using pip >= 20.3, which added support for the universal wheels before pip installing.
Install works on my M2 Apple Silicon machine! 🎉 I think this issue can be closed now.
apparently Github Actions now have beta M1 public runners.
Might I make a suggestion? Feel free to use FlyCI's M1 and M2 runners. Our runners are on average 2x faster and 2x cheaper than GitHub's AND we have a free tier for OSS projects (see below).
Install Instructrions
Easily replace your M1 runners:
jobs:
ci:
- runs-on: macos-latest
+ runs-on: flyci-macos-large-latest-m1
steps:
- name: 👀 Checkout repo
uses: actions/checkout@v4500 mins/month Free for Public Repos
SInce your repo is public, then FlyCI offers 500 mins/month of free M1 runner usage with the flyci-macos-large-latest-m1 runner.
Best Regards,
Kiril Gantchev
CEO and co-founder of FlyCI