SNOW-1023077: JDBC driver package is too big

[NathanEckert]

Snowflake account identifier: FZLBQHX.CW96174 (See ticket 00686664 for more data and context)

Hello,
We are having issues with the size of the jars, despite the use of the new experimental thin-jar #1554 (comment).
Indeed, we are packaging our application to PyPi , and there is a default size limit of 60 MB to avoid any abuse.

We are having some issues with the Snowflake jar sizes:

• 3.13.30 (last one we were able to package):
  • 45 MB total (including our Snowflake specific code)
  • 31 MB from snowflake-jdbc-3.13.30.jar alone
• 3.14.5:
  • 78 MB total (including our Snowflake specific code)
  • 67 MB from snowflake-jdbc-3.14.5 alone
• 3.14.5-thin:
  • 73 MB total (including our Snowflake specific code)
  • 5 MB from snowflake-jdbc-3.14.5-thin (this does not include non-shaded dependencies)

The size reduction of the thin jar is only apparent, as the driver itself is still 65 MB. You can see it when creating a dummy project with the following pom.xml:

cat pom.xml
<project>
  <modelVersion>4.0.0</modelVersion>
  <groupId>not-used</groupId>
  <artifactId>fat</artifactId>
  <version>standalone</version>

  <dependencies>
    <dependency>
      <groupId>net.snowflake</groupId>
      <artifactId>snowflake-jdbc-thin</artifactId>
      <version>LATEST</version>
    </dependency>
  </dependencies>
  <build>
    <plugins>
      <plugin>
        <artifactId>maven-shade-plugin</artifactId>
        <executions>
          <execution>
           <phase>package</phase>
           <goals>
              <goal>shade</goal>
           </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

and then run mvn package and du -h target/fat-standalone.jar
Running the same with version 3.13.30 yields 32 MB only!

Side note: we encounter the issue on PyPi, but there are size constraints in other places, such as (based on the issues filed on the Snowflake repository):

• PyPi: our case
• Gitlab artifacts: #174 (comment)
• AWS Lambdas: #1554 (comment)

This issue prevents us from picking the latest version for bug fixes and new features

[sfc-gh-dszmolka]

hi and thank you for raising this issue - also for sharing the support case #id. we're taking a look.

[sfc-gh-dszmolka]

the official support case has been closed, but there's some which we would like to share here in case it helps someone else too.

The thin jar for the driver is still in experimental phase, and will be there for a while. At the same time, we thank you and everyone who adopted it so early and provided their feedback ! Based on those feedbacks and on existing plans, the development and optimization of the thin jar is still ongoing.

Something which we already confirmed is not applicable for your case, but might be applicable for someone else while the development efforts are ongoing - you might be able to achieve smaller artifact size by excluding unused dependencies.
Example with thin jar with excluded GCS and AZURE:

<dependency>
      <groupId>net.snowflake</groupId>
      <artifactId>snowflake-jdbc-thin</artifactId>
      <version>3.14.5</version>
      <exclusions>
        <exclusion>
          <groupId>com.google.api</groupId>
          <artifactId>gax</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.google.cloud</groupId>
          <artifactId>google-cloud-core</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.google.cloud</groupId>
          <artifactId>google-cloud-storage</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.microsoft.azure</groupId>
          <artifactId>azure-storage</artifactId>
        </exclusion>
        <!--        <exclusion>-->
        <!--          <groupId>com.amazonaws</groupId>-->
        <!--          <artifactId>*</artifactId>-->
        <!--        </exclusion>-->
      </exclusions>
    </dependency>

We'll keep this thread posted and you can also watch the official release notes for the JDBC driver for updates.

[leo7]

Thanks @sfc-gh-dszmolka , do we have any update for the driver size? We recently have an issue caused by the driver size as well. If not, we plan to use older version to reduce the package size.

[sfc-gh-dszmolka]

hi - no specific update besides the above; which can be used to potentially reduce the size of the artifact in the thin jar.
besides that, perhaps #1710 also worth following

[leo7]

Thanks for your comment @sfc-gh-dszmolka , really appreciated! We tried thin jar and will work around by excluding certain dependencies from the jar file.

[NathanEckert]

On a side note, why did the size of the JAR more than doubled between 3.13 and 3.14 ?