RecursionError in Python called from Java can crash the process

[niloc132]

It appears to be required that python's recursionlimit is set above the default for this to happen - binary search gave me a value of 2367 will be successful with the code below, but 2368 will fail.

Plain python, which throws an exception (note the "repeated 997 more times" doesn't match the recursion depth):

>>> import sys
>>> sys.setrecursionlimit(10000)
>>> def foo():
...   foo()
... 
>>> foo()
Traceback (most recent call last):
  File "<stdin>", line 2, in foo
  File "<stdin>", line 2, in foo
  File "<stdin>", line 2, in foo
  [Previous line repeated 997 more times]
RecursionError: maximum recursion depth exceeded

Tested in jshell:

System.setProperty("jpy.jpyLib", "...python-3.10.13/lib/python3.10/site-packages/jpy.cpython-310-x86_64-linux-gnu.so");
System.setProperty("jpy.jdlLib", "...python-3.10.13/lib/python3.10/site-packages/jdl.cpython-310-x86_64-linux-gnu.so");
System.setProperty("jpy.pythonLib", "...python-3.10.13/lib/libpython3.so");
import org.jpy.*;
PyLib.startPython();
PyObject.executeCode("import sys\nsys.setrecursionlimit(2368)\ndef foo():\n  foo()\nfoo()", PyInputMode.SCRIPT, null, null);

Note that with a slightly smaller recursion limit, we see a much better error (and with a slightly different "line repeated" value):

> PyObject.executeCode("import sys\nsys.setrecursionlimit(2367)\ndef foo():\n  foo()\nfoo()", PyInputMode.SCRIPT, null, null)
|  Exception java.lang.RuntimeException: Error in Python interpreter:
Type: <class 'RecursionError'>
Value: maximum recursion depth exceeded
Line: 4
Namespace: foo
File: <string>
Traceback (most recent call last):
  File "<string>", line 4, in foo
  File "<string>", line 4, in foo
  File "<string>", line 4, in foo
  [Previous line repeated 1021 more times]

|        at PyLib.executeCode (Native Method)
|        at PyObject.executeCode (PyObject.java:138)
|        at (#17:1)

[niloc132]

It is beginning to look like this only affects a range of Python versions, and may be fixed in 3.11+. Additionally, the segfault appears to actually start in python's own code rather than jpy's handling logic - but only when jpy is calling python (so far).

[niloc132]

Confirmed, I'm unable to reproduce this on any version of 3.11 or 3.12, only on 3.10. It seems likely that there is a way to reproduce this with plain Python, or at least without jpy, but I can't yet make it happen.

Here's the simple class I'm using right now, for future debugging:

public class RecursionError {
    public static void main(String[] args) {
        // Correct these paths for your own machine:
        System.setProperty("jpy.jpyLib", "......./python3.10/site-packages/jpy.cpython-310-x86_64-linux-gnu.so");
        System.setProperty("jpy.jdlLib", "......./python3.10/site-packages/jdl.cpython-310-x86_64-linux-gnu.so");
        System.setProperty("jpy.pythonLib", "...../usr/lib/x86_64-linux-gnu/libpython3.10.so.1");

        PyLib.startPython();
        PyObject.executeCode("import sys\nsys.setrecursionlimit(2368)\ndef foo():\n  foo()\nfoo()", PyInputMode.SCRIPT, null, null);
    }
}

[niloc132]

3.8 does not experience this crash, but 3.9 (as of 3.9.19) does.