deeplearning4j/deeplearning4j

ND4J: Add an option to only include backends for specified operating systems

Closed this issue · 5 comments

anatoliy-balakirev commented

We have a Spring Boot application, which is only used in Windows and Linux. Adding ND4J dependency as follows:

            <dependency>
                <groupId>org.nd4j</groupId>
                <artifactId>nd4j-native-platform</artifactId>
                <version>1.0.0-M2.1</version>
            </dependency>

Increases a size of the resulting jar to almost 1GB. I see a lot is coming from dependencies for Android, iOS and different Linux distributions, like for ARM, PPC, etc.

When using this:

            <dependency>
                <groupId>org.nd4j</groupId>
                <artifactId>nd4j-native</artifactId>
                <version>1.0.0-M2.1</version>
            </dependency>

The resulting jar is around 350MB, but if built on Linux - it only contains dependencies for Linux (which is expected, according to this documentation: https://deeplearning4j.konduit.ai/multi-project/explanation/configuration/backends)

When trying to add dependencies with classifiers explicitly as follows:

        <dependency>
            <groupId>org.nd4j</groupId>
            <artifactId>nd4j-native</artifactId>
            <classifier>linux-x86_64</classifier>
        </dependency>
        <dependency>
            <groupId>org.nd4j</groupId>
            <artifactId>nd4j-native</artifactId>
            <classifier>windows-x86_64</classifier>
        </dependency>

Nothing seem to work at all, as the loader in the Nd4jBackend contains 0 entries:

        ServiceLoader<Nd4jBackend> loader = ND4JClassLoading.loadService(Nd4jBackend.class);
        try {
            for (Nd4jBackend nd4jBackend : loader) {
                backends.add(nd4jBackend);
            }

Excluding unneeded dependencies from the nd4j-native-platform doesn't work either, as all of them got the same group / artifact ids and Maven doesn't support exclusions based on classifiers.

Is there a way to specify native backends for only a selected subset of operating systems?

agibsonccc commented

@anatoliy-balakirev during the build just specify -Djavacpp.platform=$YOUR_PLATFORM. This will avoid bundling a bunch of dependencies. See more here: https://deeplearning4j.konduit.ai/multi-project/how-to-guides/developer-docs/javacpp

anatoliy-balakirev commented

@agibsonccc thanks for your response. Sorry, maybe my question was not clear enough. What I want is to build one jar, supporting both: linux-x86_64 and windows-x86_64. I.e., something similar to <artifactId>nd4j-native-platform</artifactId>, but with limited amount of platforms. The option you provided doesn't seem to support that.

treo commented

When switching to using classifiers, I guess you had just the classifiers in there?
You would still need to keep the classifier-less nd4j-native dependency.

anatoliy-balakirev commented

@treo thanks, that was indeed an issue. Adding this to the pom does the trick:

        <dependency>
            <groupId>org.nd4j</groupId>
            <artifactId>nd4j-native</artifactId>
            <version>1.0.0-M2.1</version>
        </dependency>
        <dependency>
            <groupId>org.nd4j</groupId>
            <artifactId>nd4j-native</artifactId>
            <classifier>linux-x86_64</classifier>
            <version>1.0.0-M2.1</version>
        </dependency>
        <dependency>
            <groupId>org.nd4j</groupId>
            <artifactId>nd4j-native</artifactId>
            <classifier>windows-x86_64</classifier>
            <version>1.0.0-M2.1</version>
        </dependency>

P.S. Not sure if it should be as part of this issue, but would be great to update documentation with that info.

agibsonccc commented