go-python/cpy3

add support for multiple Python3 versions (not just 3.7)

christian-korneck opened this issue · 14 comments

christian-korneck commented

Currently only Python 3.7 is supported. This is because the Python3 C API slightly changes between versions. For example Python 3.8 removed the PyEval_ReInitThreads function, so binding it fails. (Also sometimes new functions get added).

We should add support for multiple Python versions. This should happen in a way that we don't just delete the functions that aren't supported in newer Python versions anymore (as this would be a breaking change for users of older Python versions).

If breaking changes are unavoidable, this should lead to a new major version.

lsm1103 commented

Whether you can use the factory class design pattern to make it compatible with existing Python versions to add functionality to Python 3.8 and above

christian-korneck commented

I’ve been thinking about two “UX” options so far:

  • a factory function that can be used like
python311 := cpy3.NewPython("3.11")
python311.PyInitialize()
  • multiple package imports, most likely with a split between the Python3 core API (which stays stable for all of Python3) and a version specific package on top.
import (
	py3core "github.com/go-python/cpy3/core"
	py311 "github.com/go-python/cpy3/3-11"

)

What are your thoughts and preferences? Would appreciate any suggestions and opinions from everyone. Thanks!!

nathantfrank commented

I think the second option of multiple package imports leans closer to pragmatic go.

jshiwam commented

Is anyone working on this issue? I want to take this up.

christian-korneck commented

@jshiwam I’m only doing experiments so far. Feel free to open a PR to collaborate.

  • 👍1
pwaller commented

I don't see how you could do the cpy3.NewPython("3.11") without linking in lots of cpython versions, unless I'm missing something. I'm not sure i'd want more than one python linked in! I think it would have to be separate import paths.

(And also a big +1, I now need python3.8 and I'll want more in the future!)

jshiwam commented

@pwaller Here only one version of cpython would be linked. My understanding is that the Core API would be implemented by the parent, and the child classes will have the implementation of specific versions. If you want to use python3.11 then you would do cpy3.NewPython("3.11")

jshiwam commented

@pwaller @christian-korneck here is a POC that I have created. Please take a look at it. This POC takes the second approach of having submodules for different versions rather than Using a factory function.

https://github.com/jshiwam/cpy3x

I have implemented really few apis currently. Let me know if this looks good, and could be developed into something that we have suggested in this issue.

I have tried linking multiple interpreters and it seems to work too.
Here is the commit : jshiwam/cpy3x@d5263f6

  • 👍1
jshiwam commented

Hi Everyone, I am building the generic version of this library for python3.


//CallFunctionObjArgs : https://docs.python.org/3/c-api/object.html#c.PyObject_CallFunctionObjArgs
func (pyObject *PyObject) CallFunctionObjArgs(args ...*PyObject) *PyObject {

	if len(args) > MaxVariadicLength {
		panic("CallFunctionObjArgs: too many arrguments")
	}
	if len(args) == 0 {
		return togo(C._go_PyObject_CallFunctionObjArgs(toc(pyObject), 0, (**C.PyObject)(nil)))
	}

	cargs := make([]*C.PyObject, len(args), len(args))
	for i, arg := range args {
		cargs[i] = toc(arg)
	}
	return togo(C._go_PyObject_CallFunctionObjArgs(toc(pyObject), C.int(len(args)), (**C.PyObject)(unsafe.Pointer(&cargs[0]))))
}

I don't understand why we have used MaxVariadicLength, the cpy3/object.go file has this comment which says the following:


//MaxVariadicLength is the maximum number of arguments that can be passed to a variadic C function due to a cgo limitation
const MaxVariadicLength = 20

I just wrote a sample program to check if that's the case currently or not. The following CGO program can take more than 22 arguments.

#include<stdio.h>

int Area(int a, int b, int c, int d, int e, int f, int g,int h, int i, int j, int k, int l, int m, int n, int o, int p, int q, int r, int s, int t, int u, int v){
    return a+b+c+d+e+f+g+h+i+j+k+l+m+n+o+p+q+r+s+t+u+v;
}

And it works without any issues. Was it a limitation before when this library was built? Should i change it in the new version?

M-Quadra commented

I tend to use

github.com/go-python/cpy3/v8
github.com/go-python/cpy3/v9

for 3.8 or 3.9

christian-korneck commented
  • 🎉1
darshan-verkada commented

@christian-korneck any update on Python 3.8 support ?

WillMeng commented

any progress? @christian-korneck

christian-korneck commented

not at this time, apologies. I’m exceptionally busy with something else. However, to avoid misunderstandings, this project is not abandoned and I’m looking forward to make this change.

  • 👍1