Missing ctypes
Closed this issue · 7 comments
Cool project!
I notice a couple of other tickets with not dissimilar issues. Tryng to add pandas to the requirements.txt inside internal directory, then call it returns
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/var/folders/g3/czp3xq2n1bg45g76lsl8mp1r0000gn/T/go-embedded-example-e0d15aada55c9244/pandas/__init__.py", line 22, in <module>
from pandas.compat import is_numpy_dev as _is_numpy_dev # pyright: ignore # noqa:F401
File "/var/folders/g3/czp3xq2n1bg45g76lsl8mp1r0000gn/T/go-embedded-example-e0d15aada55c9244/pandas/compat/__init__.py", line 18, in <module>
from pandas.compat.numpy import (
File "/var/folders/g3/czp3xq2n1bg45g76lsl8mp1r0000gn/T/go-embedded-example-e0d15aada55c9244/pandas/compat/numpy/__init__.py", line 4, in <module>
from pandas.util.version import Version
File "/var/folders/g3/czp3xq2n1bg45g76lsl8mp1r0000gn/T/go-embedded-example-e0d15aada55c9244/pandas/util/__init__.py", line 8, in <module>
from pandas.core.util.hashing import ( # noqa:F401
File "/var/folders/g3/czp3xq2n1bg45g76lsl8mp1r0000gn/T/go-embedded-example-e0d15aada55c9244/pandas/core/util/hashing.py", line 24, in <module>
from pandas.core.dtypes.common import (
File "/var/folders/g3/czp3xq2n1bg45g76lsl8mp1r0000gn/T/go-embedded-example-e0d15aada55c9244/pandas/core/dtypes/common.py", line 27, in <module>
from pandas.core.dtypes.base import _registry as registry
File "/var/folders/g3/czp3xq2n1bg45g76lsl8mp1r0000gn/T/go-embedded-example-e0d15aada55c9244/pandas/core/dtypes/base.py", line 24, in <module>
from pandas.errors import AbstractMethodError
File "/var/folders/g3/czp3xq2n1bg45g76lsl8mp1r0000gn/T/go-embedded-example-e0d15aada55c9244/pandas/errors/__init__.py", line 6, in <module>
import ctypes
ModuleNotFoundError: No module named 'ctypes'
panic: exit status 1
goroutine 1 [running]:
main.main()
name := "example"
ep, err := python.NewEmbeddedPython(name)
if err != nil {
panic(err)
}
lib, err := embed_util.NewEmbeddedFiles(data.Data, name)
if err != nil {
panic(err)
}
ep.AddPythonPath(lib.GetExtractedPath())
cmd := ep.PythonCmd("-c", "import pandas\ndf=pandas.DataFrame()\nprint(df)")
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
err = cmd.Run()
if err != nil {
panic(err)
}I am on OSX, this SO answer says need to reinstall python with a library available. Can you suggest how to do that with the embedded version of python here?
I wondered if it could be related to using an old pip?
ERROR: Could not find a version that satisfies the requirement ctypes (from versions: none)
ERROR: No matching distribution found for ctypes
[notice] A new release of pip is available: 23.0 -> 23.0.1
[notice] To update, run: /tmp/python-pip-darwin-amd64-macosx_11_0_x86_64/bin/python3 -m pip install --upgrade pip
panic: exit status 1
goroutine 1 [running]:
main.main()
Would you be able to give instruction on how to update python/pip so ctypes/pandas/matplotlib etc are usable? be a nice example as these are really common python libraries
I removed ctypes due to it not being supported on MUSL builds, which go-embed-python currently uses to ensure that it works on Alpine based docker images. Changing this would require more work in regard to platforms support, so that MUSL and GLIBC builds are supported at the same time.
thanks @codablock thats a bit of a gamechanger then!
Is it possible to use go build tags perhaps so that for docker/your uses its the default, but for those (me etc) who are running Go locally or whatever, we can add a build tag and get ctypes?
That way your situation wouldn't get affected and no breaking changes for you and the rest of us who want ctypes can do
go build -tags=ctypes
or something similar?
Mind you not sure 'where' you removed the ctypes - but the idea is you don't have to support "at the same time" it could be an either or?
I've been looking for this kind of thing for years and it seems your as close as anyone. This would be greatly appreciated option!
In the meantime I have gone back a commit as i see it in the history - but would prefer not to fork off here.
Sperately, have you considered embedding everything into the app? I was wondering about whether hosting jupyter notebook from a Go binary was possible...
Is it possible to run a pip install at runtime? Like can I call pip install XXXX as a line of python in a variable at runtime rather than everything being in requirements and downloaded during go generate?
Stumble upon the same issue during my Windows builds. +1 for the resolution, but going to fork as well to roll the changes back.
@amlwwalker Working with build tags is the way to go in the future, so that all Python distributions, e.g. MUSL (without ctypes) vs GLIBC (with ctypes) are supported. These would however also require to rethink the release and tagging process as the size of to-be-downloaded embedded binaries gets huge.
The biggest problem here is however lack of time on my side. I started this project as I needed something like this in my main project (https://github.com/kluctl/kluctl) and right now, go-embed-python has all the necessary features to support this initial use case.
Ask if there is any new progress