Possibility to use Py_Initialize only once and use an external python script from multiple go routines
subhradip-bose opened this issue · 2 comments
My use case is we need to call an external python script with different parameters from multiple go routines. If every time we do Py_Initialize , then there might be memory overhead and performance issues will strike. Is it possible to do Py_Initialize only once and execute the python script from different go routines.
This project is just a relatively thin wrapper around Python's C API. So basically all the limitations of the Python C API apply. One limitation is that Python has a global interpreter lock (GIL). You can't do multiple things in a Python interpreter at the same time (unless you use Python's concurrency features in the Python code that you're executing).
The easiest way to make embedded Python to work with goroutines is to make sure that only one goroutine is actually "doing" Python at a time and all the other ones have to wait. Here's an excellent blog post and related conference talk that cover this and a few other options.
Having said this, something very exciting seems to be happening in the CPython project that will probably be available in Python 3.12 later this year (and can already be tested in Python 3.12 alpha / from git master): Python will get truly parallel subinterpreters.
https://martinheinz.dev/blog/97
While I haven't tried it, I think it should be possible to use these improved subinterpeters in Python code called from Go, like the examples from the blogpost. (However, you would need to make this project work with Python 3.12 alpha first (by default it works only with Python 3.7, but it's generally possible to get it to run with newer Python versions by removing the bindings for functions that don't exist anymore in the newer Python version. See my comment at the end of my blogpost here: https://poweruser.blog/embedding-python-in-go-338c0399f3d5 ).
Thanks for the nice and detailed explanation and it helped a lot.