Version the liblz4-java.so or it's path
brianhks opened this issue · 7 comments
We have a project that has two clients each one using a different version of lz4 java library. One has been shaded so the dependencies won't conflict but because the path to the .so isn't versioned they both grab the same .so and one fails.
Can you elaborate more on the problem you have? Do you run the two clients in the same JVM process?
We have code that uses both the cassandra client and the kafka client. Both of which have lz4 as a dependency but different versions. We have tried shading the jars but the problem is they both look for the same .so to extract. Even if we were able to move the .so to a different folder both lz4 libraries will try to extract the file to the same place. Both versions look for the so in
/linux/amd64/liblz4-java.so
One of them gets the wrong .so file.
As I think about it now the problem is not just with the name of the .so file. You also have to version the native entry points, otherwise you cannot load both libraries exporting the same functions.
Thanks, I see. Just in case..., can't you use two different class loaders for the two clients, instead of the shading?
Two class loaders would work, except you write the .so to the same place and one would overwrite the other. Problem is also I'm not in full control of the class loading in this particular project.
lz4-java writes the .so to a unique temporary file, so there should be no problem of overwriting.
It is easy to add a version to the liblz4-java.so path in jar, but as you pointed out, versionizing every external function is also necessary, which is a relatively large change. As I would like to quickly release a new version of lz4-java, hopefully this month, I would like to postpone considering this issue until after the next release.
Looking at this link: http://stackoverflow.com/questions/40026804/loading-multiple-versions-of-java-classes-that-use-native-code
It appears that when java loads the symbols it doesn't do them globally so it appears you can load the same library twice (having different names) and the symbols will not clash. So I think all you have to do is change the name of the .so and you should be good.
I think it is the case only when you load the two .so's with two different class loaders, but anyway it's worth experimenting with. Let me try....
Because each class-loader maintains its own list of loaded libraries, it is guaranteed that the JNI code called for a native method will be the correct version, even if a different class loader loads a different version of the shared library.