[Bug] Cython deadlock on upon being called back with non-callee thread
junrushao opened this issue · 9 comments
Was due to the fact that non-python started thread needs to call an explicit state ensure function. Perhaps we should register such functions e.g. runtime.ThreadInit as PackedFunc from cython side
https://docs.python.org/3/c-api/init.html#non-python-created-threads
@tqchen this is interesting finding! Just curious, is it required by only Cython, or both ctypes and cython?
I am not too sure, ctypes's mechanism might already supported this initialization
Related issue: apache/tvm#7919
Hmmm...okay, it is not so related...
It is related since we can introduce EnvThreadInitialize in the same registry as a followup
Looks like we have to somehow link to Python to make sure the following snippet could run:
PyGILState_STATE gstate;
gstate = PyGILState_Ensure();
/* Perform Python actions here. */
result = CallSomeFunction();
/* evaluate result or handle exception */
/* Release the thread. No Python API allowed beyond this point. */
PyGILState_Release(gstate);It might be possible with cython if we register them into two packed functions, but I don't have much idea about ctypes :-(
@zxybazh Looks like we can actually register these APIs with ctypes: apache/tvm#7919 (comment)
Something related: A viable solution for Python concurrency
It’s a fancy story, which won’t become a real solution in short term