[java] Generate self-contained jar with OS-Arch specific native libraries (JNI, Core and Backend)

[vkuzmin-uber]

Feature

Support fat, fully self-container Neuropod API jar with OS-Arch specific resources that has JNI lib, Neuropod Core and TF, Torchscript backends.

Note that currently Bazel build has target neuropod_jar_jar that already produces similar Jar with JNI lib and Neuropod Core. But it places it to top level and this is why it has either Linux or Darwin files only.

Also currently we don't put TF and Torchscript Backends into jar files and it relies that Backend files can be found during "backend registration". We have a use cases when we want to have single Jar that has Tensorflow and Torchscript backends and use it as "default" backends (current backend registration allows "default" and also multiple versions if available at NEUROPOD_BASE_DIR (/ust/local/neuropod by default).

We want to improve it and support Linux and Darwin both. It should work the following way,
JNI and Neuropod Core

• if build target at Linux, put JNI, Core files to path to "com/uber/neuropod/native/linux-x86_64"
• if build target at OSX put files to "com/uber/neuropod/native/darwin-x86_64"
• LibraryLoader should check what is current OS/Arch and unpack appropriate JNI and Neuropod Core only (currently it doesn't check for OS/Arch).

Backends (versions are defined by NEUROPOD_TORCH_VERSION and NEUROPOD_TENSORFLOW_VERSION)

• Add backends packages as tar.gz packages to to path to "com/uber/neuropod/native/{OS}-{Arch} directories (Neuropod repo has appropriate tar.gz targets that has version).
• LibraryLoader should check what is current OS/Arch and unpack embedded Backends to NEUROPOD_BASE_DIR (or /usr/local/neuropod if not set).

This way produced Jar will be self-contained for current OS-Arch and already can be used to run Java tests.
Later we will have "release" script that can add appropriate files for other platform.

Is your feature request related to a problem? Please describe.

Michelangelo has NeuropodJNITransformer that uses Neuropod Java API and neuropod model. It runs "offline" inference in Spark in Yarn cluster where application is distributed as jar file, self-contained fat Jar is necessary in this case.

Describe alternatives you've considered

• Not supporting Spark at Yarn but Spark at Peloton only.
• Separate neuropod Jar for Linux and OSX

In both cases, we looked at existing TensorflowJNITrnasformer that uses official Tensorflow jar "libtensorflow_jni.jar" that is self-contained and had native libraries for Linux, OSX/Darwin and even Windows.

Additional context

This is not a Uber specific request, but sort of common practice. Bazel project uses it to pack Linux, Darwin and Windows embedded JREs. Tensorflow repo supports libtensorflow_jni.jar with Darwin, Linux and Windows binaries inside.

Note that Bazel build currently doesn't have a recommended way to do it and every project has its own workaround. For instance Tensorflow uses Maven to produce final jar.

[VivekPanyam]

I think this issue/feature request is combining two ideas that would be useful to think about separately:

Cross-platform jar files that contain binaries for both Mac and Linux

As you mention, several projects do this (including TF). This makes sense to me and is not specific to your usecase.

Including backends in the jar files

This part feels a lot more specific to your usecase and I think we could solve this in a better way by decoupling it from the problem above.

You mention

Michelangelo has NeuropodJNITransformer that uses Neuropod Java API and neuropod model. It runs "offline" inference in Spark in Yarn cluster where application is distributed as jar file, self-contained fat Jar is necessary in this case.

as the reason for wanting to include backends in the jar file, but I think this usecase may be better solved by an "installer" jar that you ship as part of your application rather than pulling this functionality into Neuropod.

As I mentioned in #458 (comment), the point of the recent backend loading refactor was to avoid shipping backends in a package for each language Neuropod can be used from (e.g. pip packages, fat jars, etc). Instead, one standard way of installing backends (tar -xf ./some_backend.tar.gz -C $NEUROPOD_BASE_DIR) will make them available to run models from all languages Neuropod supports (C++, Python, Java, Go, etc).

One of the things that #458 does is bundle backends into the final jar file and untar them at runtime. It seems like that introduces functionality that is not needed for many usecases and might fit better in an internal "backend installer" library than in Neuropod itself.

[vkuzmin-uber]

Yes, I thought about separate Jar as installer. I will check if it fits our use case.

I will publish separate PR for part 1, os-specific that includes Core native libs only. I will close this one after that.