Consider using JPype

JPype was just released as version 1.0.0

Maybe this could be an alternative to PyJnius? What would be the respective benefits/drawbacks?

See also the discussion at kivy/pyjnius#551.

WOW. @Thrameos is a true champion, herding all the cats. 🏆

Thanks very much @imagejan for raising this issue. Especially now, since we will start the push for 1.0.0 very soon.

We will need to give JPype a serious look now and see whether it can fully replace our usage of PyJnius. Here is my (incomplete) list of requirements off the top of my head:

Binaries available on conda-forge for all major platforms (✅ )
Complete transparent wrapping of Java-side APIs to Python, such that you can simply import Java classes in an intuitive way and treat them as Python classes
Conversion of major data structures between Java and Python (that's where this library can continue to help, especially for our community's image data structures, but also collections etc. as needed).
- Ideally automagically so callers don't have to write
```
result = from_java(javaObject.doAThing(to_java(my_python_object1), to_java(my_python_object2)))
```
  but instead merely
```
result = javaObject.doAThing(my_python_object1, my_python_object2)
```
  See also kivy/pyjnius#457.
Ability to implement Java interfaces on the Python side (maybe not a hard requirement, but we currently use it e.g. to register event listeners from Python so the Python side knows about certain events like Java-side stdout and stderr message). (IIUC ❌ but it's coming in the next major version being developed)
Ability to start Python from Java (❌ But PyJnius also cannot do this; IIUC JPype's next major version will enable this, which will be a huge win for our community, allowing people to call CPython code directly from ImageJ! 💯 )
On macOS, loads the JVM via the libjli hack or similar to avoid the dreaded "No Java Runtime present, requesting to install" dialog (will need to test this—if it doesn't work, maybe can file a PR to JPype, but I expect they solved it long ago)
Deals with AppKit main thread nicely on macOS—i.e. painless ability to pop an AWT/Swing and/or JavaFX GUI without hanging (see imagej/pyimagej#23)

And if we're lucky, switching to JPype would magically fix other issues such as imagej/pyimagej#34 too...

Here is a run down. (Not trying to poach, but just to answer the questions.)

Anaconda binaries
Transparent Java to Python, use from java.lang import String, import java or mypkg=JPackage("com.mypkg") depending on how much you would like to expose.
Conversion of data structure. Two options on this front. You can either define a customizer or define converter methods or a mix of both. (pyjnius has some of this with dunder methods but it isn't wired up with annotations)

If you just want to make a java object have an altered method use

@JImplementationFor(MyClass)
class MyClassImpl:
    @JOverride  # This will rename the original method to have an underscore
    def doAThing(self, arg1, arg2):
           return from_java(self._doAThing(to_java(arg1, arg2))

But this can be painful if there is too many interfaces so the convert style may be better.

@JConverter(MyJavaInput, implementof=PythonClass) 
def convert(cls, obj):
    return to_java(obj)

@JImplementationFor(MyReturnType)
class MyReturnTypeImpl:
    # insert the methods that make the class a duck type for so that you dont have to call from_java but instead expose the Java class as if it were a Python one.
    ...

This has the advantage that MyJavaInput will match PythonClass everywhere it is used. And because the return type is a duck type there is no need to convert to Python on the return. So if you have a huge interface with many methods taking the same arguments this would be best.

Implement Java interfaces from Python. Easy

import java
@JImplements(java.util.function.Consumer)
class MyConsumer:
   def apply(self, arg):
       pass

Start Python from Java and call Python methods from Java. Sorry next version.
MacOS. Not sure my coworkers use JPype all the time, but perhaps this is solved.
AppKit... I believe this is covered. There is an old file where someone worked out the patterns for this. I personally haven't used it (and was debating throwing it away as it was mostly a 5 liner). We can run multi-threaded code including GUI code as far as I am aware, but the limitations on being able to extend classes from within Python sometimes mean some adapter library glue code is needed.

@Thrameos Thanks so much for your answer!

Regarding Python from Java... it's not super urgent—but will be awesome for our community once it happens. The fact that you are working on it is very much appreciated.

About macOS: I'll be testing on macOS for sure, so will let you know what I find out. And if I know how to fix, will file PRs as needed. Hopefully some time in August.

There is one other hugely important requirement for us, which I forgot to mention: shared memory. We use pyjnius together with the Unsafe API on the Java side to wrap Python-side memory pointers in Java, so that we can access NumPy images with zero copying. Because our image library (ImgLib2) is interface-driven, all we (@hanslovsky) needed to do was write an alternate off-heap/Unsafe-backed container implementation for ImgLib2, pass the memory pointer to Java from Python, and wrap it. This has been a huge boon for people combining ImageJ with the NumPy world, because bioimage data often reaches many GB in size these days.

Do you think this approach will continue to be possible JPype?

JPype provides direct linkage between shared memory using direct nio buffers that can be created using any unsafe memory. These can be created in Python, Java, or externally. There are some implications about lifespan depending on the source. If it has to survive JVM shutdown to should be Python or externally created. This is all detailed in the userguide. I believe it was part of the 30 pulls in 30 nights campaign thought it may have been earlier. These can be converted to bytes, bytearray, or memoryview on the Python side. They can also be mapped to numpy arrays subject to not shutting down the JVM. So I think we have that need covered.

Great, thanks again. I'll do my due diligence in the next 1-2 months and explore in more detail!

Does JPype handle overloading methods when implementing Java interfaces? I know that PuJNIus does that.

Currently JPype has classes that implement an interface use a dispatch approach rather than directly implementing each overload. I can add individual overloads as a feature request if it is considered important, but no one has ever requested it.

Example:

public interface SomeInterface {
    public void doSomething(int a);
    public void doSomething(int a, Object o);
}

@JImplements(SomeInterface)
class MyImpl:
    @JOverride
    def doSomething(self, *args):
         if len(args)==1:
             # Called as int
         elif len(args)==2:
             # Called as  int, Obj
         else:
              raise MethodNotFoundException("No overload")

The reason for this is that when calling from Python, we will hit the dispatch as individual overloads are not available.

If I had to implement individual overloads I would likely do the following.

@JImplements(SomeInterface)
class MyImpl:
    @JOverride(JInt)
    def doSomething(self, a):  # This would get name mangled to "_doSomething(I)"
         # Call from int

    @JOverride(JInt, JObject)  # This would get name mangled to "_doSomething(ILjava_lang_Object;)"
    def doSomething(self, a):
         # Call from int, Object

    # The new method would need to scan for matching overloads and attempt to construct a dispatch function so that incoming requests get routed properly.

Is that what you are looking for? Do you consider this a necessary feature?

Disclaimer: I am not working on this anymore, so my opinion should be considered food for thought. Ultimately, the active maintainers need to decide what feature would be desirable to them. I defer to @imagejan and @ctrueden to make a decision on this but maybe you find my thoughts helpful:

I did not think of the *args approach but I think it is a clever idea. I do have one concern with it:
Overloading through conditions on the input always feels a little messy compared to typed and compiled languages. Many times, just checking the number of arguments will not be enough but you also need to check the types of the arguments. Programmers with a more exclusive Python background probably have a very different opinion on this.

I really do like the way that PyJNIus does it:

If a method has only a single overload: Match by name
If there are multiple overloads, specify the method name and signature in a decorator

In my opinion there are two downsides to this: Programmers need to know how to specify the signature and you have two specify all overloads, even the overloads use the exact same semantics (e.g. passing int or float to a function). In the JPype case you could simply do if isinstance(args[0], (int, float)).

Summary:

JPype overloading

Pros:
- need to write only one single method, straight forward for Python developers
Cons:
- "overloading" by checking argument count and types can be messy, especially if you usually program in a language that supports overloading

PyJNIus overloading

Pros:
- Be explicit about method signature & let compiler/framework do the dispatch
Cons:
- Need to implement/specify all signatures, even if they use the same implementation logic
- Need to know how to specify Java signatures (might be a barrier for Python developers and probably some Java developers, too)

I personally like the way it is handled in PyJNIus but that may not be generally true for the target audience: Maybe they prefer the more Pythonic way of JPype

Another thing to consider: Does JPype auto-convert Java arrays into Python lists? PyJNIus does that and I am not a huge fan of that (I would prefer a JArray object or similar). It is not uncommon, in Java, to pass an array as output parameter. This is impossible to implement on the Python side if the array is copy-converted into a Python list.

Proxies

Yeah that pretty much captures it. Though I can likely support both styles if that is necessary. If you don't put any arguments to @JOverload it would simply bind by name as a dispatch. If it does see arguments then it would write its own dispatch which does a delegate to Java.

User writes

@JImplements(MyInterface)
class MyImpl:
    @JOverride(JInt)
    def method(self, i):
        pass
    @JOverride(JFloat)
    def method(self, i):
        pass
 
    @JOverride
    def method2(self, arg1, arg2):
        pass

The class factory writes

@JImplements(MyInterface)
class MyImpl:
    def "_method(I)I"(self, i):
        pass
    def "_method(F)F"(self, i):
        pass
    def method(self, *args):
        return JObject(self, MyIterface).method(*args)  # Send to Java for dispatch.

    def method2(self, arg1, arg2):
        pass
    "_method2(I,I)I" = method2
    "_method2(F,F)V" = method2

The prestep would add all the bindings to the class with the signature name (adding the return type). The Python call would then be allowed to delegate through Java if there are overloads specified or directly depending on the implementation.

As far as the implementation. Java just passes the name and the method signature name to a Python function which is responsible for deciding which to call. I can change it very easily. The proxy compiler is actually doing method stealing. It gets each class and then traverses methods copying methods and member to a fresh class adding the required mark up for Java.

Arrays

JPype has a very different concept of arrays. JPype only alters the return type for methods in 3 special cases. null=>None, true=>True, false=>False. This is a requirement because those objects are singletons and can not be overloaded. In all other cases, JPype will return a wrapper object which is duck typing to the nearest Python type. JChar appears to be a length one string with math operations, JInt (and related) are a class derived from int, JFloat( and related) produce a class related to float, and Arrays become a Sequence (called JArray) implementing class which is backed by a Java array.

That means you never pay the cost of converting unless explicitly requested. In some cases it is actually more work to send the wrapper, but we do that so that when you chain Java calls, the return type is tracked so it picks the correct overload for the next call. (Much of this was inherited before I picked it up, but that was intended as a system to resolve overloads, and I extended it to cover the return types.) We had the same issue as pyjnius because the original author had made strings convert and that lead to make it impossible to chain certain string operations. I deprecated it and transitioned the project to use wrapper only style over the last year which is why we are finally at 1.0.

As mentioned before my local user group that I am supporting are a bunch of physicists so programming is not their specialty. Minimizing the number of rules and making everything duck type so they are not aware which is in Python or in Java was a goal. It does mean the occasional rough spot where you have to insert np.array(ja), list(ja) or str(jstr) to force the type where you want a conversion to take place, but otherwise it is pretty clean.

We do auto convert the other way. dict => HashMap, list=>ArrayList, path=>File, path=>Path, datetime=>Instant, etc. You can abuse it to make some pretty slow running code, but for setting fields once or simple calls these help a lot to make the code appear clean.

This sounds pretty good and I think it's definitely worth a closer look before the scyjava/imglyb/pyimagej 1.0 release.
Maybe it could even resolve the reference counting issue the we observed with PuJNIus: kivy/pyjnius#345

Specific overloads sound good but it's probably best to evaluate if it's worth the complexity. I like it but I will likely not use in the foreseeable future.

Specific overloads is about 50 lines of code changed.

Add a handling to proxy to check if the proxy has specific or general overloads.
Mangle the name if specific.
Pass method name (or mangled to Java entry point)
Add name mangling to @JOverload.
Check if any overloads are requesting name mangling and pass it to JProxy construct so that Java can pick up the managling.
Add dispatch for any methods with name mangling.
Add alias for any named methods that do not have mangling enabled.
Add test cases for name mangled and un-mangled methods.
Revise the documentation for mangling.

This stuff is on the long term task list mostly because much of it is on the list for providing extension of Java classes from Python. But that item has three or four other items in the form of prerequisites in front of it. So if you don't plan to use it, I will likely postpone until the other item comes up (at which time the effort cost will be reduced as all related prerequisites will have been completed.)

So if you don't plan to use it, I will likely postpone until the other item comes up

Just wanna make sure that you don't make a decision based on my comment that I won't be using it: I won't be using it because I am not working in the scyjava realm anymore. @ctrueden and @imagejan might have other plans

Don't worry. I put it on the road map for JPype 2.0. I can always push it forward if someone feels strongly that it is beneficial, or I can keep it there until it is trivial to implement in the future. Usually I will get it inside of a year, unless it is something that requires a huge effort. About the only item that I missed in the last big push was PyPy support and that is only because I was unable to locate someone who was able to help me with debugging.

Ok, I wonder what would be the changes needed here to switch from jnius to jpype.

Is there an exact replacement for jnius.autoclass(...)?
Is there an equivalent for jnius_config?
Can we replace:

scyjava/scyjava/__init__.py

Line 140 in c593029

jnius = _init_jvm()

... by something like:
```
jpype.startJVM(os.path.join(workspace, '*'))
```
?

@imagejan I'll be doing a deep dive later this month (with @elevans), and will report back here.

The nearest replacement for autoclass is jpype.JClass . It creates a class wrapper from a string. It is one of 4 methods one can use to import a class.

By name JString = jpype.JClass('java.lang.String')
By imports . import jpype.imports; from java.lang import String as JString
As an attribute of a package. JString = jpype.JPackage('java').lang.String
Or converting a class instance to a wrapper from Java. Class=jpype.JClass('java.lang.Class'); JString=jpype.JClass(Class.forName('java.lang.String')) The example is a bit contrived as I had to create the class instance using forName. But this last one does see use when you have to manually set up class loaders, get a class by reflection, or need to debug exactly where a class loaded is failing on something like a missing shared library.

Options to the JVM are passed to jpype via startJVM. There is a minor difference in philosophy. In PyJnius, configuration happens before PyJnius meaning the JVM must be started, while in JPype the import does not imply the JVM is started until requested. Because of this different, JPype code can have things such as decorating the customizers happen before the JVM.

In PyJnius we load the config, set up the options, then import to start

import jnius_config
jnius_config.add_options('-Xrs', '-Xmx4096')
jnius_config.set_classpath('.', '/usr/local/fem/plugins/*')
import jnius

would be this in JPype where we import, then call startJVM to launch.

import jpype
jpype.startJVM('-Xrs', '-Xmx4096', classpath=['.', '/usr/local/fem/plugins/*'])

For the most part JPype provides similar API to pyjnius with some differences to note.

The conflict symbol for names is a trailing underscore in JPype, so if in Java it is 'Foo.class' in JPype it would be 'Foo.class_'. I believe PyJnius is the reverse.
JPype with only 3 exceptions returns a Java type not a Python one. So if you need to force a Python type such as to call a Python string method apply the converted. PyJnius s=obj.callMethod().startswith('fred') would need to be written as s=str(obj.callMethod()).startswith('fred') The exceptions are Java null, true, and false which map to "None", "True", and "False" respectively. (Sorry I can't derived them on Python side as they are singletons, not that it hasn't stopped me from trying.)
Arrays are handled very differently. PyJnius always converts arrays on return and requires arrays to come in as lists. JPype instead provides a duck type Sequence for Java arrays. Thus you control when data is moved from Python to Java and back. If some data needs to be manipulated a lot on the Java side then just pass around the Array handle rather than getting it back. JPype does this for multidimensional arrays and buffers as well.

There is one other somewhat significant philosophical difference in how JPype functions from other bridges. That is the differences between light and heavy wrappers. Many projects really don't expose the use of Java at all. They are Python libraries that just happen to use Java on the backend to do work. Those types of libraries hide the Java from view. This "heavy" wrappers require coding all the API functionality that will be exposed in Python and maintaining it and the corresponding documentation. You can always use JPype as a backend just like the other bridges.

My goal in JPype is to expose Java packages as Python packages entirely. This means that a Java package should "be the Python module". If you say want to add Python functions the Java package "org.scijava" you will simply place a org/scijava/__init__.py inside of the jar file (or another jar that is loaded with it). So to use a JPype library, you just have the user start JPype with the jar file on the class path and then import org.scijava and the module code will appear with all the extra stuff that Python needs. You can add wrappers to increase the functionality of Java classes. So for example Java provides a Thread concept, but to use it from Python we need extra methods to attach and detach. (The embedded python code in Jar feature is in the PR queue for the next feature release.)

@_jcustomizer.JImplementationFor('java.lang.Thread')
class _JThread(object):
    """ Customizer for ``java.land.Thread``

    This adds addition JPype methods to java.lang.Thread to support
    Python.
    """

    @staticmethod
    def isAttached():
        """ Checks if a thread is attached to the JVM.

        Python automatically attaches as daemon threads when a Java method is
        called.  This creates a resource in Java for the Python thread. This
        method can be used to check if a Python thread is currently attached so
        that it can be disconnected prior to thread termination to prevent
        leaks.

        Returns:
          True if the thread is attached to the JVM, False if the thread is
          not attached or the JVM is not running.
        """
        return _jpype.isThreadAttachedToJVM()

    @staticmethod
    def attach():
        """ Attaches the current thread to the JVM as a user thread.

        User threads that are attached to the JVM will prevent the JVM from
        shutting down until the thread is terminated or detached.  To convert
        a daemon thread to a main thread, the thread must first be detached.

        Raises:
          RuntimeError: If the JVM is not running.
        """
        return _jpype.attachThreadToJVM()
...

The intent being to support "light" modules in which the cost of maintaining the Python module is minimal as Java interface (and documentation) serves 90% of the API needs and Python is just needed to add that extra glue like this extra functionality to java.lang.Thread

PyJnius does this under the hood to deal with collections types, but they use hard coded maps so the option of exposing a Java library with full customization isn't really achivable. JPype is more intended to compete in the space of Jython where try mixing of Python and Java code is needed.

I am working on giving some finer grain controls to the method dispatch system as well, so that you can just declare the adapters and converters that apply to a class or method rather than needing to define them globally. But that is still being fleshed out.

Edit: Not that I am trying to rule out the use of JPype as a backend for a Python library as sometimes people really do want to exposed a different API in Python; it is just the JPype goal is to eliminate having to write a whole additional Python API a requirement to use a Java library. The extra time saved can be used to improve the Java library which benefits everyone.

Thanks for sharing in so much detail @Thrameos I think that these are really strong points in favor of JPype

Starting the JVM on import jnius (or scyjava, imglyb, pyimagej etc) has always been a drawback rather than a feature in my eyes

Hi everyone -- here's a quick summary of the work done so far to migrate pyimagej from pyjnius to jpype.

scyjava

JPype migration status: Complete

Working status: Working

All tests run successfully with JPype.

Structural changes to scyjava

Because JPype initializes the JVM like this:

import jpype
import jpype.imports

jpype.startJVM()

And not through an import like pyjnius (e.g. import jnius) I restructured scyjava to account for this difference in JVM control. I added an additional module scyjava.jvm that contains everything related to the JPype JVM. To do this I just adapted the pre-existing scyjava to work with JPype and made sure that everything higher in the stack points back to the scyjava.jvm module. jnius_config has been removed and is no longer needed.

The JPype version of scyjava is on the jpype-test branch.

imglyb

JPype migration status: Complete

Working status: Working

Migrating imglyb was straight forward (mostly). I did not make any significant structural changes.

The JPype version of imglyb is on a repo attached to my account here (I did not have rights to create a branch on the imglyb repo proper).

pyimagej

JPype migration status: In progress
Working status: mostly working, debugging

There are now a two differences between the pyjnius and jpype version of pyimagej. The imagej attribute .py now needs a leading _. For example ij.py.to_java(result) needs to be ij._py.to_java(result). The legacy attribute also needs to have a leading _. For example ij.legacy().isActive() is now ij._legacy.isActive().

So far I've replaced the old pyjnius code with JPype and I'm debugging a few issues. Right now I just hit this particular issue with ambigous overloads (there are probably but I haven't found them yet).

Ambiguous overloads (example from the pyimagej jupyter notebook tutorial):

If you try to run a macro like this:

macro = """
#@ String name
#@ int age
#@ String city
#@output Object greeting
greeting = "Hello " + name + ". You are " + age + " years old, and live in " + city + "."
"""
args = {
    'name': 'Chuckles',
    'age': 13,
    'city': 'Nowhere'
}
result = ij._py.run_macro(macro, args)

I recieve an Ambiguous overloads error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/edward/Documents/repos/loci/pyimagej/imagej/imagej.py", line 319, in run_macro
    raise exc
  File "/home/edward/Documents/repos/loci/pyimagej/imagej/imagej.py", line 316, in run_macro
    return self._ij.script().run("macro.ijm", macro, True, to_java(args)).get()
TypeError: Ambiguous overloads found for org.scijava.script.DefaultScriptService.run(str,str,bool,java.util.LinkedHashMap) between:
	public default java.util.concurrent.Future org.scijava.script.ScriptService.run(java.lang.String,java.lang.String,boolean,java.util.Map)
	public default java.util.concurrent.Future org.scijava.script.ScriptService.run(java.lang.String,java.lang.String,boolean,java.lang.Object[])

I've also uploaded the environment.yml file for my conda env for conda users. To get this working you'll need to install the dev versions of scyjava and imglyb. I'm also using JDK8 (I had to set this manually with my $JAVA_HOME variable as conda will complain about conflicts if you attempt to install openjdk-8 via conda).

The JPype version of pyimagej is on the jpype-test branch.

My system: Ubuntu 20.04.1 LTS
Mentions: @ctrueden @hinerm

Is there a restriction in JPype that is forcing the leading underscore for py and legacy? Is it something that I need to address?

Is there a restriction in JPype that is forcing the leading underscore for py and legacy? Is it something that I need to address?

The py and legacy attributes are Python-side additions to a wrapped-from-Java object. To quote the docs:

JPype only allows the setting of public non-final fields. If you attempt to set any attribute on an object that does not correspond to a settable field it will produce an AttributeError. There is one exception to this rule. Sometime it is necessary to attach addition private meta data to classes and objects. Attributes that begin with an underbar are consider to be Python private attributes.

We'd prefer not to use a leading underscore, since those attributes are intended to be part of the public API. But we didn't see an obvious way of doing that.

@elevans wrote:

TypeError: Ambiguous overloads found for org.scijava.script.DefaultScriptService.run(str,str,bool,java.util.LinkedHashMap) between:
	public default java.util.concurrent.Future org.scijava.script.ScriptService.run(java.lang.String,java.lang.String,boolean,java.util.Map)
	public default java.util.concurrent.Future org.scijava.script.ScriptService.run(java.lang.String,java.lang.String,boolean,java.lang.Object[])

@Thrameos Is there a way to tell JPype which method we want? Our issue here is that we have a java.util.LinkedHashMap that we want it to match the java.util.Map signature, but there is also a java.lang.Object... (i.e. varargs) signature that also matches (in that the map could be made into a single-element Object[]). When using this API from Java, the Map signature has higher precedence and is used without conflict.

There are two options that you can use to deal with setting fields that are not allowed.

First when you create a class customizer you are allowed to set any field including taking over Java methods and redirecting them specific implementations. Using @JImplementationFor to define the prototype class which will serve as a back bone for the class when it first is constructed. There is also a special method called __jclass_init__ which is called not just for the class which is being customized, but all classes that derive from the class. This can be used to make major modifications to existing classes wrappers.

So lets do an example... here I am going to add an attribute called py with the value of Fred to every instance of java.lang.String

import jpype
import jpype as jp
from jpype.types import *
from jpype import JImplementationFor
jpype.startJVM()

@JImplementationFor('java.lang.String')
class JStringProto(object):
    @property
    def py(self):
        return "Fred"

js = JString("hello")
print(js.py)

This incidentally how you fix problems like LinkedHashMap overload. You override the troublesome method which will rename the original method to have a leading underbar and then implement a new python method use interfaces to decide which is the best overload. Then you use the casting operator to select the specific overload. That said I will look into the the overloading. It would seem like the map would have precedence, but I suspect it is other one of these cases where both ended up with the implicit level. I really need a derived level which matches better, but that is more work and potentially breaks some compatiblity.

The other option is to directly set an attribute telling it to bypass the setter. There were changes in Python 3.8 and later that make this option more difficult to do so I am not sure if this will be a good long term option. We used to use this sort of monkey patch often.

import jpype
import jpype as jp
from jpype.types import *
jpype.startJVM()

js = JString("hello")
type.__setattr__(js.__class__, 'py', 'Fred')
print(js.py)

Hope that helps.

I should also note that I have in the past been working on improved customization of methods. For example, JPype used to support a global switch on whether methods should return Python strings or Java strings. Such a global switch makes it hard to use specific Java methods which require chaining, but that doesn't mean that for a specific class it wouldn't make sense to covert strings for methods.

To give finer grain control I proposed adding annotations to the customizer class which allow a class, field, or method to specify a type to be used for returns in the form of a map of converters. For example, if the class converters were given as Java String=>str(x) then every method for that class would produce a string automatically (without breaking the basic Java classes). We could add "rename" to a specific overload or other customization options so long as they are easy to use as simple markups using decorators. This is towards my long term goal that Java classes should be easily customized to fit the requirements of Python so people don't need to write heavy wrappers.

I am in the process of releasing JPype 1.1 and starting the next cycle. If you have any requests for the next so that I can address in the next release now would be a good time. I assume that the ImplementationFor pattern solved the last issue.

@Thrameos Thanks, with your example I was able to get the .py and .legacy attributes set properly! I've been playing around for the last couple of days and I haven't gotten the @JImplementationFor pattern to work with the ambiguous overloads issue yet. I'll take any help/comments you've got.

I tried to work up an example of the ambiguity that you were seeing but I was unable to formulate anything in which a linked hash map would have problems with Object[]. It would seem more likely that it was a static vs member overload problem.

Example:

import java.util.Map;

public class Test {
        public int call(Object[] j) {
                return 1;
        }

        public int call(Map j)  {
                return 2;
        }
}

And

import jpype
jpype.startJVM(classpath=".")                                                                                                                                               Test = jpype.JClass("Test")
print(Test)
t= Test()
print(t.call([1,2,3]))
print(t.call({"a":1,"b":2,"c":3}))
print(t.call(jpype.java.util.LinkedHashMap({"a":1,"b":2,"c":3})))

Gives "1,2,2" as expected without exceptions for JPype 1.1.

So the first question is can I see the definitions of the method with an issue, the line that caused the issue, and the type of each of the arguments?

If for some reason I forget the object instance then I would get something like...

Traceback (most recent call last):
  File "overload.py", line 15, in <module>
    print(Test.call(jpype.java.util.LinkedHashMap({"a":1,"b":2,"c":3})))
TypeError: No matching overloads found for Test.call(java.util.LinkedHashMap), options are:
        public int Test.call(java.util.Map)
        public int Test.call(java.lang.Object[])

It seems it is having problems with the first argument (Object[] or Map vs LinkedHashMap) when it is really having issues with argument 0. We gave it a Test class which does not fit with a Test instance. This was a common point of confusion so we modified it to make it more likely to clue the user in.

The new and improved error message reads...

Traceback (most recent call last):
  File "overload.py", line 15, in <module>
    print(Test.call(jpype.java.util.LinkedHashMap({"a":1,"b":2,"c":3})))
TypeError: No matching overloads found for *static* Test.call(java.util.LinkedHashMap), options are:
        public int Test.call(java.util.Map)
        public int Test.call(java.lang.Object[])

If this isn't the problem I would need a short example so I can look over the method resolution procedure.

@Thrameos Thanks for looking into it! Did you try with varargs Object... rather than Object[]? With a varargs signature the Map could be implicitly wrapped into a single-element Object[], hence the ambiguity.

I did not explicitly test “Object…” perhaps the variable argument it the problem here. I may be able to make a rule that variable arguments should prefer a single match if that is the problem without breaking anything else.

I ran some tests and it does appear that the varargs is part of the problem. The name resolution which constructs that method resolution table in Java does not appear place a resolution for Map over Object because it considers Object[] and Map to be separate non-derived types without considering a single Object instance. It appears to be a bug in org.jpype.manager.MethodResolution. It needs an additional clause in isMoreSpecificThan.

For those that are interested here is how to resolve this kind of issue. We have two methods which appear to be ambiguous but one has more specific types. In order to know which one to use there is a table which consults a cache of of method takes precedence over method. So we have call(Object...) vs call(Map). It should say call(Map) is more specific than call(Object). The table is built when the dispatch is created (we don't currently have a way to alter it).

Each method in the dispatch is compared against each other method in a pair wise fashion calling org.jpype.manager.MethodResolution.isMoreSpecificThan.

Fortunately these methods are all exposed to the user so we can debug it directly by manually calling the internal methods of jpype.

Test = jpype.JClass("Test")
methods = [ i for i in Test.class_.getMethods() if i.getName()=="call"]
MethodResolution = jpype.JClass("org.jpype.manager.MethodResolution")
print(MethodResolution.isMoreSpecificThan(methods[0], methods[1]))
print(MethodResolution.isMoreSpecificThan(methods[1], methods[0]))

We get the result 'False,False' so it currently has no preference between them hence a warning issued if both resolutions match. To fix that we just need to check if one of the methods is varargs and add some extra code so that it recognizes
the correct precedence order.

The finished product is in jpype-project/jpype#863

@Thrameos That's awesome, thank you so much for explaining how it works! If we encounter more ambiguities like this, we'll dig in and file a PR (or at least have more specific details if we get stuck). 🥇

@Thrameos I think your fix was correct, but actually didn't work for our specific case due to java.lang.Boolean.TYPE not being part of the conversion list.

I opened a PR to address this... hopefully Boolean wasn't excluded for a particular reason?

Nope. Likely an oversight. I merged it in. Likely there needs to be more tests in the test bench so that we can make sure things don't break. However, that should be part of a larger effort to test the MethodResolution class.

Just a quick update for lurkers: the following branches go together now:

Things are largely working thanks to @elevans's efforts. We'll keep pushing on it, with the goal of getting 1.0.0 releases of the component stack (jgo, scyjava, imglyb, pyimagej) done by mid-November.